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TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC)." 



General 



1 . At least over the nexi three years, research and evaluation activities should receive as high a 
priority as course production and programming. This will require either finding additional 
resources, or a moderate but noticeable re-direction of existing resources to research and 
evaluation activities. 

2. Staff who write papers should be encouraged to lodge a copy, including a short abstract, 
with the Librarian (who compiled the Iistin Appendix C.) This list should be kept up to date, 
and new additions to the list circulated ( Adth abstracts) twice a year to all OLI and KN staff. 

Creating a framework 

3. Each programme area or department should be responsible for evaluating its own 
performance; consequently, each programme area and department should be responsible for: 

(a) identifying and reviewing its own performance criteria, and having these accepted by 
the next senior level of management; 

(b) identifying the information it needs to collect on a regular basis in order to evaluate 
its own performance, and collecting and analysing that information 

(c) identifying and implementing special research projects which aim to enhance the 
performance of the department 

(d) formally reviewing, at least once a year, its own performance, in the form of a report 
to the next senior level of management; it is at this point that next year's performance 
criteria should also be considered and agreed. Research and evaluation data would 
be 20S of the sources of information on which this review would be based. 

(e) department and programme area heads should provide a short report as a result of 
this review to the CEO, and these reports should be discussed at the Executive 
Council. The CEO should write annually a summary report on performance for the 
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Board meeting (in May?), using both these reports and statistical information on 
course enrolments, completion rates, and trends (see recommendations regarding the 
data-base for more details). 

The OLA Board needs to disseminate to staff of both institutions the fundamental change in 
stance regarding their role following the adoption of the Mission and Goals statement; the 
implications for working style and professional development need to be explained and 
understood by staff at all levels, but especially by senior managers, who need to take positive 
steps to encourage professional development. Without this, the climate necessary for good 
quality research and evaluation (and, more importantly, for achieving the broader Agency 
goals) will not exist. 

Every member of staff should be encouraged to have a planned set of professional 
development activities over a year's period. These activities should be of benefit both to the 
individual and the Agency. It should be the responsibility of each manager to negotiate 
professional f velopment activities with staff in their department. Each person should be 
entitled to the ei ivalent of half a day per week, or 25 days per year, minimum, for approved 
professional development activities, to include personal research activities, attendance at 
conferences, training courses, visits or exchanges with other relevant institutions, overseas 
consultancies, approved courses of study, work on new developments, and other 
non-operational activities which lead to increased competence in the post. Thus the 25 days 
will include a number of activities already being carried out. This general principle should 
apply to all levels and grades of staff, although the amount of time for professional 
development activities may vary across grades. 

There are several ways in which central staff can be brought into meaningful and useful 
contact with students and tutors: 

(a) appointment of a student and a tutor representative to the OLA Board 

(b) the appointment of a paid student editor for the OLA newspaper, with a remit to 
increase the number of student subscribers and contributors 

(c) appointment of a student or tutor to selected project management teams 
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(d) establishment of a students 1 association, with elected student representatives; OLA to 
provide audio and/or computer conferencing, including the use of a students* 
bulletin board for elected representatives 

(e) provision of two or three self-operated video boxes, which record viewers 
comments, and which can be broadcast back to the audience at a later date, to 
provide direct feedback on KN programmes 

(f) occasional weekend student/tutor get-together days, within programme areas, to 
discuss general student issues (study techniques, course choice), etc., on an 
optional, voluntary basis. 

7. Two kinds of research and evaluation activities for the Authority are proposed: 

(a) regular activities (such as course evaluation) which are built into project 
management; 

(b) special projects: individual studies, commissioned or approved either within KN or 
OLI, or in conjuction with other agencies 

In either case, procedures are proposed to ensure that each type of activity is approved, 
properly resourced and professionally conducted. 

Tlie data-base 

8. 1 Improvements to the data-base along the lines proposed should receive the highest priority in 
the Information Systems development programme. 

8.2 Data from the data base should be accessed for a specific reason, related to policy or 
decision-making; this means there should bespecific times when the information is needed, 
and a specific person or set of persons who would be committed to using this information. 

8.3 Routine or regular provision of information should therefore be kept to a minimum, and 
related to particular regular decisions or policy requirements. 



8.4 Self-access should be the main way in which the data-base should be used. In other words, 
instructional designers, middle managers, and senic managers (or their assistants) need to 
be able to get information from the data-base relatively easily. Those people who need 
information should not only be responsible for defining their needs, but should wherever 
possible access it themselves, at times when they need it, and without having to learn 
specialist programming skills; advice though should be available on the best means to obtain 
the information required. Appendix D outlines some of the analyses which it ought to be 
possible to access easily and conveniently. 

8.5 It is important to define the research and evaluation needs of different groups of people and 
to develop procedures of data abstraction which meets those needs. The following needs 
have been identified: 

i Course maintenance. It will be recommended (9.5) that each course should be reviewed 
as soon as possible after the first semester presentation of that course. Thus for each new 
course, the print-out in Appendix F should be commissioned by whoever is responsible 
for course maintenance. (A programme for this already exists). If any further 
information is needed from the data-base, this should be acquired by the maintenance 
person by directly accessL.^ the data-base. 

ii Middle management (directors, course co-ordinators, possibly vice-presidents). They 
need an overall picture of the progress of courses in their areas. It is recommended that 
this is necessary pnly once a year, rather than semester-by- semester, as at present. Once 
a year, middle management should get the print-out outlined in Appendix G, covering a 
ful1 y ear period, i.e. foi all students registered for each semester within a specified 
year-period. All course maintenance people should also automatically get a copy, so they 
can see how their individual courses compare with others. 

ffi Executive Council and the Board The Executive and the Board need an annual picture 
both as a form of accountability, and to help inform discussions regarding future policy 
for the Institute/Authority. A statistical summary (as outlined in Appendix H) therefore 
should be provided to the Board and Executive Council. The annual course statistics 
should cover a full financial year, to make it easier to relate expenditure to performance. 
This summary should be the main document, but Appendix G would also be available to 
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the Executive Council and Board members, for those who want to look more closely at 
the figures. , 

8.6 Where it is not possible to access easily the required information, or where a detailed or 
complex investigation is required, this should be treated as a project, with the consequent 
need for approval and allocation of suitable resources, and setting of priorities. These 
enquiries would entail the use of a specialist to obtain the necessary information, and 
possibly special computer, programmes to be written. 

8.7 The existing tools for or ganising and extracting data are not adequate for allowing easy and 
convenient self-access for the purposes of undertaking statistical analyses. What is required 
is a programme which automatically merges new student data with existing student data, in a 
consistent way for all students, to create a dynamic, up-to-date master student file. 

8.8 Once this basic master student file is created, standard statistical analysis software packages 
(such as SPSS) can be bought, which will enable data to be easily abstracted in the form 
required by anyone authorised to do so, whether or not they have programming skills. It is 
recommended that such a package capable of running on the mainframe should be purchased. 

8.9 For research or evaluation enquiries requiring substantial statistical analysis or cross-breaks, 
the appropriate information required (e.g. information about students on a particular course) 
should be abstracted from the master data-base for local processing on micros, thus reducing 
CPU time. This would require purchasing one suitable micro for each main programme area 
(i.e. approximately five) plus suitable micro software packages, to provide statistical 
analyses and cross-breaks. Thus simple analyses could be run directly on the mainframe; 
more complex analyses would be done on micros, to reduce CPU time. 

8.10 In order to provide accurate costing and an efficient implementation of these 
recommendations, Information Systems should be requested to do an initiation study to see 
what is required to provide a basic, dynamic master student file, and what packages would 
be most appropriate for abstracting and analysing data, including standard statistical tests. 

Course evaluation 

9. 1 Course evaluation should be considered as a component of the project management process 



for course design and delivery, for every new course. 

9.2 Courses should be designed with the firm intention of running for at least six semesters 
without changes; in most cases, changes after, that date would be restricted, and limited 
wherever possible to one set at a date and within resources agreed m the planning stage; in 
other words, the aim is to design courses which are viable from the first dat» of presentation 

9.3 The following course evaluation decisions need to be made at the project management 
planning stage: 

(a) discussion of overall evaluation strategy, followed by a decision regarding whether a 
standard or special evaluation approach should be adopted for the course 

(b) evaluation costs, work-load and responsibilities, and the timing of the evaluation 

(c) the implication for resources, production, delivery and tutoring of any changes likely as 
a result of evaluation 

9.4 A standard evaluation approach would ensure that every new course is reviewed at a set time 
soon after first presentation. Details of how this is done can be refined and possibly 
simplified, but one suggested procedure is as follows: 

(a) every new course should be reviewed within six months of the first semester of 
presentation (or as near as possible to that time, if examination arrangements prevent 
results being available within that period) 

(b) collection and analysis of information for the course review should be seen as part of the 
maintenance of a course (i.e the management.of a course during its presentation); one 
person (i.e. the person responsible for course maintenance) should be responsible for 
co-ordinating evaluation information 

(c) ah evaluation file for each course should be opened from the start of course design : into 
this file would go course developer's doubts and queries about design, course 
consultant's comments, delivery and tutoring issues, errors spotted after delivery, letters 
and phone-calls from students, advisers and tutors, and any emergency action taken; the 
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file should bfrkcpt open for-the whole life of a course; 



(d) emergency action would be taken within the first six months of course presentation only 
in exceptional cases (i.e. where serious consequences likely to lead to widespread failure 
would result because of noi acting). 

(e) the person-responsible for course maintenance should commission within six months of 
the first presentation of the course: 

i a standard print-out of the first semester enrolments and grade distributions (see 
recommendation 8.6:i.) 

ii tutor statistics (see recommendation 12.3) 

iii. a statistical analysis of assignment and examination questions as soon as they are 
available; these can be compared with data from other comparable courses; if 
necessary, more data (e.g. student demographics) can be called up from the 
data-base (recommended changes to the data-base will make this procedure much 
simpler and easier than at present). This information would be added to the 
evaluation file. 

(f) The senior tutor or course co-ordinator should formally contact tutors (and exam, 
markers if different) by phone after the first examination for detailed comments on the 
course, using a semi-structured set of questions, designed in conjunction with the course 
developers. The senior tutor or course co-ordinator would then write a report on the 
course based on this tutor feedback. This report would be added to the evaluation file. 

(g) the senior tutor or course co-oidinator would then prepare a report on the course, using 
all the information in the evaluation file, including a recommended course of action. A 
copy would be sent to the head of the programme area. 

(h) if the course is considered to be generally satisfactory, or changes can be accommodated 
within the maintenance budget, the evaluation file would be kept open by the course 
maintenance person, who would also be responsible for drawing attention to any 
subsequent developments which might require unplanned changes to the course; 
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(j) if there are major problems, which cannot be easily explained from the standard 
evaluation (e.g. an unusually high drop-out rate), the head of the programme area would 
decide whether to bid for funds for a special evaluation; whether to leave the course as it 
is; or whether to withdraw the course altogether 

(k) where changes are being proposed to an existing course, tutors should be consulted 
about the proposed changes 

These recommendations need not be followed in detail, but provide an example of how a 
standard evaluation procedure might be conducted. 

9.5 A special evaluation of a new course may on occasion be appropriate, but as an exception 
rather than a general rule. This would involve a more detailed study of a course, and would 
usually be linked to a prior commitment to substantial changes to the course as a result of the 
evaluation, if this proves necessary. Special evaluations, because of the cost, should be used 
selectively. One procedure would be as follows: 

(a) If a special evaluation is decided at the initial planning stage, it should be timed so that 
information is collected during the first and possibly second semester, otherwise, 
information should be collected as soon as possible after a standard evaluation has 
identified problems. It will usually be necessary then to run a course for several 
semesters before a revised version can be introduced. 

(b) Special evaluations should use postal questionnaires and/or telephone interviews for 
students, and any informal group discussions that can be arranged, as well as the 
procedures outlined for standard evaluations. 

(c) Once the information is collected, decisions should be made about necessary changes, 
within the budget allocated; in addition, an executive summary should be widely 
circulated and an open seminar held to discuss the implications of the report 

9.6 Piloting or pre-testing of courses should be treated as a special evaluation. 

9.7 It is recommended that each major programme area should review its activities annually. This 
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would be a meeting of the staff within a programme area, chaired by the head of the 
programme, which would look at courses, delivery, tuition and other issues. The aim of the 
review would be to identify necessary changes to next year's programme of activities, and 
might coincide with budget projections for the following year. For this meeting, a paper 
should be drawn up which provides a summary of enrolment and grade information for all 
courses in the programme area over the year period (see recommendation 8.6.ii.). Course 
evaluation reports prepared during the year would support this documentation. 

Project research 

1 0. 1 Evaluation should be built into project management. This means: 

(a) discussing the evaluation strategy at an early project planning stage 

(b) determining at this stage the necessary resources (manpower, funds, and timing of 
evaluation) 



(c) conducting evaluation as a defined, continuous activity throughout the project 

(d) defining criteria for success 



i.2 Evaluation designs for development projects should be flexible, to reflect the context of the 
project, and the often exploratory nature of development projects; initial objectives for 
instance may legitimately be revised as a result of experience 

.3 It is important in the evaluation of development projects to look at process as well as 
outcomes (how best to do things, as well as looking at costs and learning results). This may 
require: 



(a) a qualitative, observational approach 

(b) identification and recording of key events in a project 

(c) explanation of why events/decisions were made, as well as description 
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(d) examination of the teaching or decision-making process, examining, for instance, the 
nature of interaction between learners and teachers, or what kinds of J earning take place 
as a result of developmental activities 

(e) analysis of the development's impact on work roles, need for new skills, training, etc. 

(f) resource implications, not only in terms of costs incurred, but how a new development 
affects the balance of costs between different activities and spending departments (e.g. 
audio conferencing may require a shift of costs ftom production to delivery) 

10.4 For these reasons, it is recommended that evaluation of development projects should 
generally be conducted internally, i.e. by project staff, but with external assistance from a 
professional evaluator, acting as a consultant, wherever possible* This means that evaluation 
resources may better be spent increasing the internal manpower of a project, to allow internal 
staff to have adequate time for evaluation activities. Credibility will denend then on-high 
quality, clearly written evaluation reports, which allow those not directly involved to 
understand the development project, its achievements and difficulties. 

10.5 To ensure high quality evaluation, it is recommended that evaluation should account for a 
minimum of 10% of internal manpower resources on a development project. > 

11.1 All research projects should go through a formal approval procedure, to ensure projects of 
significance are undertaken, to avoid duplication and to ensure that they are competently 
conducted. 

(a) Bids era be proposed by any individual or group of staff, but should be discussed with 
their superior manager and referred upward for approval 

(b) The Executive Council should assess and if appropriate approve any single research 
proposal requiring more than $50,000 of resources (manpower and/or cash), or 
involving co-operation with external institutions, or paid for out of the Executive's own 
research and development budget 

($>i Vice-Presidents should assess and if appropriate approve any project within their area 
requiring less than $50,000 worth of resources from within that area 
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(c) The Director, Research and Evaluation Methods, should be consulted regarding the 
assessment of all research proposals (see recommendation 14.2) 

11.2 Research projects may be resourced from the following sources (see recommendations 
14.5-14.10): 

(a) re-assignment of duties of internal staff (e.g. reduction in production activities) 

(b) a sum of money held by Executive Council (recommended amount: $100,000) and/or a 
sum of money available to each of the four component areas (recommended amount: 
$50,000 each); these sums of money may be used either to buy out internal staff from 
operational duties (using the money for staff replacements), or to hire in external 
research staff 

(c) use of professional development time of internal staff 

(d) external funding or assistance from other institutions 

(e) any combination of the above. 

The aim is to provide flexible but limited sources of funding for research projects. 

11.3 In general, it is better to encourage staff to conduct research as a team rather than in isolation. 
Resources should be concentrated on a limited number of significant projects rather than lots 
of small projects. 

1 1 .4 It is recommended that the approval procedure and sources of funding for development and 
research projects should be the same. 

1 1.5 Sources of external funding, and co-operation with external agencies, should be developed 
wherever possible. University post-graduate students should be encouraged to work with the 
OLI and KN, under joint supervision, to collect data and conduct research studies. OLI and 
KN should help University staff define significant research areas suitable for study by 
doctoral students. Joint studies on common areas of interest (e.g. drop-outs, learning at a 
distance, tracer studies following the progress of students) should be carried out jointly with 



13 



other open learning and distance education institutions, through both the proposed Centre for 
Research and Education in Open Learning, and through the various Canadian distance 
education associations (CADE, WCCUDE). 

.6 However, the opportunities for such joint and external activities should not deflect OLI and 
KN from their responsibility to set their own priorities for research and evaluation, and for 
conducting their own internal research and evaluation activities. 

.7 It is recommended that the following research projects should receive top priority. I have put 
them in my own order of priority, but the actual ordering of priority should be determined by 
the Executive Council: 

i Information Systems to develop a basic, dynamic master student file, and purchase of 
packages appropriate for abstracting and analysing data, including standard statistical 
tests. This should be a top priority within Information Systems (existing) development 
budget, since without it, evaluation and research cannot really begin 

ii development of an accurate and practical means of costing OLA activities (essential for 
project management) - funded from (new) Administration research funds 

iii Knowledge Network audience research study (see proposal 12.2): from (new) KN 
research funds, perhaps linked to (v) and (vi) below 

iv at least one special course evaluation per year (as well as standard course evaluations on 
all other new courses) in each of the Open University, Open College, and KN 
programme areas (research funds from appropriate areas' research budgets) 

v profiles of the OLI student: a statistical analysis of age, occupation, education, 
motivation by programme area, and comparison with the general population of B.C.: 
using new data-base and new project officer (no other cost) 

vi market survey, to identify the public's knowledge and opinion of OLI, KN and OLA, 
and their perceived relevance to individual's needs: (new) Executive Council research 
fund 
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vii study of means to link OLA better to provincial and federal industry, commerce, 
government and voluntary organisations, to identify better their continuing and distance 
education needs, and to educate them into the potential of open learning (e.g. directory of 
agencies and contacts, high-level conference, etc*): from Executive Council Research 
fund, with staff seconded from relevant programme areas (this should be a low-cost 
project, using existing resources as far as possible) 

viii identification of obstacles to qualification through open learning; baniers to the pyramid 
of opportunity: individual case-studies, showing the perspective from the individual 
student: from Executive Council research fund - link to (ix) 

ix tracer studies: what study paths do OLA students follow; what happens to drop-outs; 
where do OLA students come from (in terms of previous educational activities)? where 
do they go to? - from Executive Council research fund 

x project to identify whether there is sufficent field support for students, and if not, what 
steps can be taken to improve student support, and in particular whether the role and 
recruitment of tutors needs to be changed, and what role other universities and colleges 
might play in providing field support to OLA students to increase student support: from 
Executive Council research fund or Administrative research fund 

xi study of specific B.C. open learning needs and the needs of minority groups: how does 
open learning look from their perspective; what are their special needs; what can be done 
for them? Some prospective groups: 

long-distance fishermen (Open College budget) 
BC people working overseas (University budget) 
lumber industry (Open College budget) 
mining industry (Open College budget) 

- really remote students (Administration research budget) 
native Indians (Open College/ABE research budget) 
handicapped students (Administration research budget) 

- unemployed students (Open College/ABE research budget) 

- English as a second language (Open College budget) 
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No doubt otherminority groups could be identified. 



xii An analysis of the reliability and validity of assessment questions (see proposal 
12.8): from Administration budget 

Student services 



12.1 Tutors should be evaluated on 



(a) their efficiency in turning round assignments 

(b) the reliability of their marking of assignments 

(c) the quality of their communication with students (comments on assignments, 
advice and help to students) 



These three criteria should be of equal importance. 

Senior tutors should be responsible for evaluating tutors in their subject areas. Where there is 
no senior tutor in a subject area, it should be the responsibility of the course-cordinator, but 
this should be only a temporary arrangement until a senior tutor post is available. The need to 
evaluate tutors should be taken into account in senior tutor workloads. 



12.3 A senior tutor should normally request a copy of at least one assignment, preferably more, 
per tutor per semester from registry. If there is cause for concern, the senior tutor should 
then request to see a sample of the tutor's next batch of assignments. It may be necessary 
as a result of such an inquiry to re-grade assignments at the discretion of the senior tutor. 
Senior tutors should also contact each tutor on a regular basis by telephone or other means. 
Unless a system like TRIX is installed, senior tutors should also be sent each semester a 
print-out of assignments submitted and marked by all tutors in the subject area. (A computer 
routine already exists for such a print-out). This would indicate turn-round time and quantity 
of assignments marked by each tutor. In exceptional cases, senior tutors may also want to 
contact students directly by phone if they are concerned about a tutor's performance. The 
main purpose of monitoring tutor marking and commenting would be to help tutors improve 
performance, rather than to assess the need to terminate employment, although that may be 
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necessary in extreme cases* 

12.4 There should be an agreed and clearly publicised student appeal procedure regarding 
assignments. The senior tutors should deal with student appeals regarding assessment of 
individual assignments. The procedure, including to whom appeals should be made, should 
be printed in the OLI calendar. 

12.5 Senior tutors, in association with their course co-ordinators, should He responsible for 
organising occasional workshops for tutors, on assessment strategies, counselling, and 
giving helpful study advice, etc. 

12.6 Information Systems should be asked to conduct an initiation study to identify the feasibility 
and costs of introducing the TRIX system. 

12.7 It is recommended that a research project is undertaken to identify whether there is sufficent 
support for students, and if not, what steps can be taken to improve student support, and 
whether the role and recruitment of tutors needs to be changed to increase student support. It 
should iook in particular at the roles that regional colleges might play in increasing local 
support for provincial distance education students. The project should involve a management 
team which would include a remote tutor, a representative of a regional college, a Programme 
Area Director or course co-ordinator, a remote student, someone from the registry, Jan 
Muirhead or Denise Hartmann, and a regional student advisor. It would conduct a survey of 
students and tutors, and would prepare financial estimates for any 
recommended changes. 

12.8 It is recommended that, as well as monitoring by senior tutors, a means of analysing the 
reliability and validity of both continuous and examination assessment questions is devised, 
to ensure consistency in the level of qualification. A small project team should be established, 
consisting of a programme area director or course co-ordinator, someone from the registry, 
someone from Information Systems, and an external consultant specialising in examination 
validity procedures, to recommend methods and procedures. 

Knowledge Network 

13.1 A survey needs to be commissioned from a leading market research or census organisation 



(possibly Statistics Canada), to identify accurately, through careful sampling, Knowledge 
Network reception capability and utilisation within the province. If possible, the KN enquiry 
should be piggy-backed on a household survey already planned by the external research 
organisation, to reduce costs. The aim would be to identify across the province the number 
of households able to receive KN via cable with and without a convenor, or direct from the 
satellite (i.e. actual rather than potential reception facilities), the extent to which those 
households utilise KN programmes, and their reactions to the KN service in general. This 
will require extremely careful sampling procedures, and will be a restively expensive 
exercise, involving several thousand households across the province. Data collected from the 
survey should be stored on the OLI computer, and collected in such a way that it is 
compatible with the OLI stude.r data-base. This survey would not need to be repeated every 
year, so should be treated as a project. 

13.2 The mailing list already held by KN, primarily for marketing purposes, could also be added 
to the market survc; information, although it will be important that the two sets of 
information do not become confused, and work will need to be done to ensure that the 
marketing list is as compatible as possible with the OLA data-base. 

13.3 One benefit from the survey in 13.1 is that it could provide a good sampling base for 
enquiries about specific programmes, since from the imtial (large) sample of people identified 
as able to receive KN programmes, smaller samples representative of the general population 
can be drawn for specific programming enquiries, and rotated to avoid over-use of certain 
households. 

13.4 KN ought to have a greater sum for project research than the other programme areas, and 
make greater use of external research organisations. (The recommended minimum figure is 
$100,000). 

13.5 Professional development time should be used to allow KN staff to direct and participate in 
externally commissioned projects. 

1 3.6 It is recommended that research be directed at key programming policy decisions over the 
next few years. For instance, research might be conducted into: 



(a) the effectiveness of live, inter-active programmes 
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(b) appropriate production styles for different target audiences (e.g. general education, 
telecourses, teleseries) in terms of perceived audience need 

(c) the impact of video-recorders on the use of KN programming (access to recorders 
by different target audiences, implications for production style, support materials) 

(d) the need for media education (how to Vead 1 television) 

(e) ways to increase student learning from television (production style, use of recorders, 
signposting) related to different programming areas 

(f) better ways to identify and select appropriate programming (relative merits of market 
research, advisory committees, how to involve wider range of agencies, etc.) 

(g) better ways to liaise with wider public (a computerised directory of agencies, use of 
subscribers/mailing lists, newspaper, open-access video-boxes, for public to record 
opinions about needed programming and support services) 

l 

(g) analysis of programming: sources, relevance to BC, spread of target groups, market 
research into demand for areas not covered, provision of programming for the 
disadvantaged 

(h) reactions to imported programming 

Some of these studies could be integrated. Research methods would include postal 
questionnaires, telephone interviews, group discussions and individual, person-to-person 
interviews, using small but carefully selected samples of actual programme viewers. Some of 
these projects would be appropriate for joint research with Universities in the province. 

1 3.7 To ensure that research is directed to meeting the needs of KN, and is properly co-ordinated 
and integrated with programming and development work, it is recommended that a small 
committee of KN staff, plus the Director, Research and Evaluation Methods, be established 
to define research needs, allocate funds, commission studies, and disseminate and act on the 
results. 
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Organisation and funding 



14.1 (a) rough estimates should be made of person-days required to produce a course, across 
the various job categories (this is necessary in any case for proper project 
management) 

(b) each individual in the course design and production area should negotiate with their 
immediate manager a rough estimate of how time will be spread over various work 
activities; these estimates should take account of all the demands on people's 
work-times (course maintenance, administrative activities, evaluation, professional 
development, etc.) 

(c) commitment to new course production would take account of person resources 
available, and this would influence project planning; once committed, courses 
should be produced according to the agreed schedule, and staff, having agreed the 
time necessary, would work to those norms. 

14.2 It is recommended that a Director, Evaluation and Research Methods, should be appointed, 
with the following responsibilities: 

(a) advice to CEO and Executive Council on evaluation and research matters, including 
the feasibility and quality of research proposals and evaluation procedures within 
OLI/KN. 

(b) the design and contracting of evaluation and research activities commissioned by the 
Executive Council 

(c) organisation and analysis of statistical data about students and courses for the 
Executive Council 



(d) 



advice to staff in OLI/KN and other B.C. open learning agencies on evaluation and 
research methodology, including advice on the design and conduct of research and 
evaluation activities, appropriate external research agencies, sources of funding, 
report writing, and dissemination 



/f 
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(e) the co-ordination of evaluation and research activities, to avoid duplication and to 
ensure widespread dissemination of research and evaluation findings within OLI/KN 

(f) liaison with other open learning agencies involved with research and evaluation, 
with a special responsibility to disseminate research findings from other institutions 
of relevance to OLI/KN. 

14.3 The appointment of a Project Officer (Evaluation and Research), to assist OLI/KN staff in 
using the data-base, to draw samples, and to process and analyse research data; the project 
officer would be responsible to the Director, Evaluation and Research Methods 

14.4 Three options are offered regarding the organisational location of the Director, Evaluation 
and Research Methods, and the Project Officer: 

i Directly reporting to the CEO, in the same way as other Authority-wide services, 
like the International Office 

ii Location within the Administration component 

iii Creation of a new component (Authority Services) which would combine all the 
Authority-wide services, including evaluation and research assistance 

My own preference is for (i) in the short-term, but (iii) in the long term. I am not happy 
about (ii), because the Director should be independent of operational units. 

14.5 (a) The Executive Council should set priorities for research or special evaluation 

projects requiring more than $50,000 in resources (manpower and/or cash, 
including a cost for time of existing staff assigned to work on the research project); 

(b) Vice-Principles should be free to approve evaluation and research activities within 
their own area costing less than $50,000 per project, provided they have the 
resources within their areas, and have checked with the Director, Evaluation and 
Research Methods, regarding the feasibility and quality of the proposals and lack of 
duplication with activities in other areas. 

14.6 Standard evaluations should not require extra resource, The procedure recommended is not 
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labour-intensive and should reduce the overall time spent on changes to course material. 

14.7 If the recommendation regarding }. . essional development time is accepted (recommendation 
5), this will provide a substantial pool for approved development, special evaluation and 
research projects, although development, evaluation and research will not be the only 
activities carried out in professional development time. There could though be perhaps a drop 
in course production of around 5-8% per year (maximum) to accommodate this 
recommendation, even allowing for some efficiency savings from the standard evaluation 
procedure and better time management, unless extra resources are found. 

14.8 This leaves three options with regard to the recommendation regarding professional 
development time: 

(a) don't do it 

(b) see it as a replacement activity for some other activities 

(c) cost it as an extra resource requirement 

14.9 Revised procedures for making the data-base more accessible and purchase of appropriate 
software should be considered part of Information Systems normal development budget. If 
though other demands of equally high priority mean that these revisions would be 
delayed, extra resources (probably not in excess of $60,000), should be found from the 
1987/88 budget. 

14.10 In addition to a slight reduction in course production activities, these recommendations 
require the following extra resources per annum: 

Salary and overheads for Director, Evaluation and Research Methods $60,000 (?) 



Salary and overheads for Project Officer (Evaluation and Research; 



$40,000 (?) 



Cash for special evaluation and research projects: 

Executive Council $100,000 

University $50,000 

Open College $50,000 

Administration $50,000 

Knowledge Network $ 1 00,000 



Total $450,000 



Expenditure of £450,000 should enable all the proposals and projects suggested to be implemented. 
Implementation 

15. The following sequence and priorities for implementation of the proposals are recommended 

1 . Alteration t o the data-base structure . If not implemented, most of the other proposals 
cannot be implemented, either. 

2. Appointment of Director and Project Officer Essential for advice on standard 
evaluation, project research, and supporting evaluation activities 

3. Tutor evaluation Could begin immediately, if senior tutors agree 
These three proposals could be initiated immediately 



4, Standard evaluation procedure put into place Dependent on establishment of project 
management and data-base changes 



5. 



Project research Some development projects could not wait, but major research 
projects should await appointment of Director and establishment of data-base 
requirements 
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1. Introduction 



Both the Open Learning Institute and Knowledge Network are relatively young, innovative 
organisations. Also, compared to open learning organisations in many other countries, OL1 and 
KN are compact, each with small numbers of staff but large mandates. In the early days of OLI and 
KN, all efforts were understandably concentrated on course and programme production, in order to 
provide as quickly as possible a wide range of programmes and courses. The production and 
delivery of materials left little time for systematic research or evaluation. However, both OLI and 
KN are now relatively well established, and there has been a growing recognition that the two 
organisations might benefit from a planned approach to evaluation and research. 

I was therefore approached in May, 1986 by Ian Mugridge, then Academic Dean at OLI, and asked 
if I would be available to ass^oS the research and evaluation needs of OLI and KN, and to advise on 
what steps might be taken to establish an active research and evaluation programme. Consequently, 
a contract for three months' consultancy was issued, to commence in January, 1987. 

2, Terms of reference 

The full terms of reference are attached as Appendix A. The original suggestion was that the 
consultancy should be concerned with proposals for both research an£ development. However, 
between the time of the first approach and the commencement of the consultancy, a major re- 
organisation had taken place, including the appointment of a new Chief Executive. By the time I 
arrived, plans for development activities were already under way. It was therefore agreed to restrict 
the present consultancy to institutional research and evaluation. 

It was also recognised that the consultancy should take into consideration not just the requirements 
for reseaiwh, training and professional development within OLI and KN, but also the possibility of 
an Agency-wide service. Partly for this reason, it was intended that I would work with a committee 
to be formed, under the chairmanship of the Principal of North Island College, to look at research, 
training and professional development in open learning in British Columbia. This committee was 
not formed however in time for me to work with it, although I was able to liaise with an initiative 
from the University of British Columbia, concerned with the possible creation of a collaborative 
centre for research and education in open learning within British Columbia. 

Lastly, it was agreed to commit up to 20% of my time during the consultancy to other activities, at 
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the request of individual departments. 
3 . Method of conducting consultancy 
Step 1: Identifying Needs 

Having agreed the terms of reference, the first step was to interview staff in both institutions, 
usually individually, starting as far as possible with senior management. Using a semi-structured 
interview, the aim was to discover the following: 

a. what on-going or routine institutional research and evaluation activities are being, or 
have been, carried out, and how satisfactory these have been 

b. what individual initiatives had been taken to conduct evaluative studies as part of 
professional development 

c. what particular research projects had been conducted 

d. what research information staff still required 

e. what, if anything, prevented staff from conducting research and evaluation 
enquiries at the moment 

f . what priority they would give to research and evaluation activities, even if it meant 
reducing other activities to accommodate them 

g. what expertise staff had in the research/evaluation field 

h. what views staff had on the funding and organisation of research and evaluation 
activities 

i. any other comments they wished to add about research and evaluation activities 
In addition to individual interviews, meetings were also held with groups of staff, as follows: 

course designers in the University programme 
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course designers in the Open College programme 
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tutors in the University programme 
tutors in the CTV programme 
A list of staff interviewed is attached as Appendix B. 

A library search was also conducted to identify papers or articles by staT of OLI and KN, and 
papers held in the library by external writers regarding the OLI (a full bibliography is attached as 
Appendix C). 

In order to understand the nature of the student data-base, its structure, and its suitability for 
research and evaluation activities, an exercise was undertaken, with the help of Jocelyn Calvert, 
Course Co-ordinator, University Programme, and Heather Drugge, Publications and Statistics 
Clerk, to abstract information to determine what happened to students who dropped out of their 
first University Programme course (i.e. to find out whether they re-registered for another course, 
etc.). The results of this exercise are attached as Appendix D. 

Interviews were also held with several staff from the Information Systems department, to identify 
what steps were necessary and feasible to make it easier to use the data-base for research and 
evaluation activities, and discussions were held with other staff to try and identify a strategy for 
accessing and analysing information on the data-base. 

I also attended two meetings called by the University of British Columbia to discuss the 
establishment of an Institute for Research and Education in Open Learning, and the ideas and 
approac v agreed in these discussions have been taken into account in this report. 

Also, during this phase, a work plan and schedule, and the structure of this report, were detailed 
and agreed with the President-Designate of OLA. 

Step 2: Formulating Proposals 

Following this intensive period of interviewing, a number of proposals for the organisation and 
implementation of research and evaluation activities were formulated. These were put forward-to 
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those previously interviewed for consideration and comment. Occasionally, this was followed up 
with a second interview. 



Secondly, I organised and conducted a series of seminars on research and evaluation (see Appendix 
E for a list of topics). One of the purposes was to help staff understand better institutional research 
and evaluation activities and the strengths and limitations of evaluative research. However, an 
equally important objective of the seminars was to get reactions to my initial proposals. 

As a result of these discussions, some proposals were modified or dropped, and other new 
proposals emerged. Finally, this report and a draft executive summary have been prepared and 
circulated to the five Executive Council members for their consideration. 

4. Do OLI and KN need an institutional research and evaluation programme? 

What is institutional research? 

First of all, to avoid misunderstanding, it may help to define briefly what I mean by evaluation, 
institutional research, and development. 

Evaluation is an everyday activity that goes on all the time. Every time we say we like a programme 
or course, we are making an evaluative judgement. Such evaluation though can be on purely 
subjective grounds, without any evidence of effectiveness. 

Institutional research is an attempt to provide more systematic information on the performance of an 
institution. It is an attempt to collect evidence of effectiveness, to identify problem areas, and, 
where possible, to suggest alterations that are likely to lead to improvement. This kind of research 
then can be one input to evaluative judgements. This consultancy is primarily concerned with the 
need f ~r, and the means to provide, institutional research for evaluation purposes. 

Development is any new activity or new way of doing something. It may or may not be 
systematically evaluated. Research may or may not suggest new developments. Thus development 
can take place without research, although if possible, research and development should be linked. 

Relationship between institutional research, policy and decision-making 

For institutional research to be useful, it needs to be linked to policy and decision-making. There is 

30 
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no point in carrying out research which cannot be acted on. For instance, there is little point in 
discovering that most tutors have been born outside British Columbia, if that is considered 
irrelevant to hiring staff. Sometimes, research not originally directed at policy or decision-making 
may turn out to be useful, but given scarce resources, such a random approach to institutional 
research is difficult to justify. 

Secondly, institutional research on its own should not dictate policy. For instance, research may 
indicate that recent immigrants have much higher drop-out ra s than other students. However, the 
policy that follows from such research is likely to be influenced by many other factors, as well as 
the research results. 

Lastly, research may help formulate or influence policy, but cannot substitute for it. In some 
institutions difficult decisions have been postponed by calling for more research, when more 
research was not the answer, but choosing between two unwelcome alternatives. 

Nevertheless, well conducted and timely research can be a valuable, sometimes essential, 
contribution towards skilled decision-making, thus improving considerably the effectiveness of a 
distance teaching system. I have therefore looked at research in terms of policy and 
decision-making, and not in terms of an interesting but not immediately applicable academic 
exercise. 

How necessary is 1R at OU and KN? 

By definition, in a distance education system, stude.,*s are not in direct contact with course 
designers and managers of distance education institutions. There is a real danger therefore that the 
institution unwittingly adopts policies and procedures which are not effective, or which could be 
substantially improved. The very nature of distance education requires a conscious effort to bridge 
the gap between students aix< staff. Institutional research is one means by which this gap between 
students and staff can be bridged. 

Another reason is to do with the technological nature of distance education. It is at heart an 
industrial system of manufacturing, delivering and 'servicing' a product, in this case learning 
materials. The principle of feedback is considered essential for any technological system to run 
efficiently. Institutional research and systematic evaluation are means of providing this feedback. In 
other words, if open and distance learning in British Columbia is to be a self-improving system, 
institutional research is essential. 



9 

ERJC 



32 

8 



These are general arguments which would apply to any open and distance learning system. With 
respect to the Open Learning Institute and Knowledge Network, almost everyone I interviewed 
expressed the view that systematic evaluation and research had not by and large been conducted, 
and were desperately needed. The view was frequently expressed that while in the early years it had 
been essential to 'get the show on the road 1 , the time had now come to take a much more critical 
look at policies and procedures that had often developed in an unplanned and ad hoc manner. 

Many individuals felt that they were working in a vacuum; others expressed considerable unease at 
the effectiveness of certain practices, but had no hard evidence to support that unease; others felt 
that they did not really know enough about the students to be confident of designing effective 
materials. It is worth noting that some tutors complained that while they had fed back information 
about defective course material, the information had not been acted upon, because of lack of 
resources. Nearly all those interviewed were enthusiastic about the possibility of introducing 
research and evaluation activities, on a planned and professional basis. 

Both OLI and KN are in fact in urgent need of research and evaluation activities. Both institutions 
would be hard pressed to demonstrate objectively their efficiency and effectiveness, because of lack 
of suitable information. Methods of operation have become crystallised without any evidence of 
their effectiveness, or considered assessment of alternatives. The lack of research and systematic 
evaluation procedures leaves both institutions politically vulnerable, and inhibits the development of 
both institutions as leaders in open learning and distance education. However, such research must 
be related to effective policy and decision-making, and the issue of adequate resources to act on 
evaluation activities also needs to be addressed. 

Recommendation 

1 . At least over the next three years, research and evaluation activities should receive as high a 
priority as course production and programming. This will require either finding additional 
resources, u a moderate but noticeable re-direction of existing resources to research and 
evaluation activities. (Resource implications are detailed later - see Section 13.) 

Research and systematic evaluation though should not be just short-term activities. They need to be 
built into the system. This requires closely examining management structures and practices within 
the two organisations, analysing ways in which research and evaluation activities can be fitted into 
the main decision-making processes, and identifying where management procedures need to be 



adapted to encourage this. 



5. What exists already? 

The short answer is: not a lot, but still more than most staff are aware of. 
Open Learning Institute 

There already exists a comprehensive and effective system of course evaluation in both the Dental 
and the Nursing Programmes in the Open College. It is interesting to note that systematic evaluation 
is a renuirement of these two programmes for them to receive accreditation. These evaluations use 
very little in the way of resources, being conducted entirely by the respective course co-ordinators. 
These two programmes therefore provide a model of what other courses could do, although both 
co-ordinators complain of lack of resources for revising materials and for conducting more 
thorough analyses of the data collected. 

Also, a semester-by-semester print-out is supplied by Registry of course enrolments, withdrawals, 
grade-distributions, and examination performance, and of tutor success rates and performance. 
This information however is not systematised or summarised, and many course co-ordinators and 
designers complained of the difficulty in using this information in the form in which it is provided. 
A common complaint was that it was produced often too late to be of much practical use. Part of the 
problem is that it takes a great deal of computer central processing time to run the programme and 
print-out. 

Occasionally, special research studies have been conducted. Appendix C provides quite a large list 
of papers on the work of the Open Learning Institute and Knowledge Network, most of which are 
locateH in the Open Learning Institute library. I am reasonably certain that while this contains most 
of the papers available, it does not include them all. More could be done to disseminate what 
already exists. 

Recommendation 

2. Staff who write papers should be encouraged to lodge a copy, including a short abstract, 
with the Librarian (who compiled the list in Appendix C.) This list should be kept up to date, 
and new additions to the list circulated (with abstracts) twice a year to all OLI and KN staff. 



There are several reasons why papers and publications are important. The first is to disseminate 
within the institution the results of research and evaluation studies, with the intention of influencing 
policy* However, the relatively few research studies that have been conducted, while of interest in 
their own right, and with often important policy implications, were rarely tightly linked in with the 
decision-making processes, and consequently have had little impact (It is worth noting that there is 
already a report on institutional research and development prepared by a colleague of mine from the 
British Open University - Woodley, 1983). 

Another reason for publication is for professional development. The need to record, analyse and 
evaluate practices usually results in a more realistic assessment of the strengths and weaknesses of 
current procedures and policies, and may well stimulate changes in practice. 

A third reason for publication is because both OLI and KN are primarily innovative educational 
institutions, and therefore have a responsibility to disseminate knowledge about their practices and 
their effectiveness to the educational world at large. Publication is one of the more important ways 
through which the reputation of an open learning institution is judged. The great majority of the 
papers in Appendix C are descriptive of OLI and KN procedures and systems, rather than research 
or evaluation studies, or even critical analyses. There are less than 30 published articles, which is 
very little over a period of seven years for two such innovative institutions, although Mugridge and 
Kaufman's Distance Education in Canada is a major publication in the area. It is also worth noting 
that only six external studies are located in the OLI library, and only two of these were student 
assignments or theses. There is clearly a great opportunity for post-graduate student studies related 
to open learning that could also be useful to OLI and KN. 

Over all, then, OLI's and KN's publication record is not impressive, and certainly does not do 
justice to the innovative work being carried out by the two institutions. This is partly because very 
few hi2 u quality evaluative research studies have been conducted at OLI and KN. 

6. Some comments on research methodology 

It is customary to think of research as a highly scientific, quantitative and precise exercise, based on 
carefully controlled experiments, testing hypotheses. This is ons. method of research which has 
proved its value in the physical sciences. Certainly, a good deal of institutional research - on 
enrolments, drop-outs, demographics of students, sampling, for instance - is quantitative, and 
requires good statistical techniques. 
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However, institutional research is worthless if the results arrive too late, are too narrow to help 
decision-making, are unintelligible to decision-makers, or are hedged around with so many 
qualifications that no course of action seems better than any other. Institutional research is always a 
compromise between the ideals of empirical, scientific research on the one hand and the demands of 
decision-making and the constraints of time and money on the other. Very often, institutional 
research is more useful if it is quick and dirty, rather than clean and late. Better to be 75% certain, 
and act on the information, than 90% certain after the decision has been made. 

More importantly, for an evaluation to be useful, it is often necessary to understand the context in 
which decisions have to be made, ard the underlying reasons why decisions get made. This 
requires a much more qualitative, interpretative and usually participative approach to research, 
using careful observation and interviews, and drawing on personal experience, in an attempt to 
understand what is happening. This is particularly important for development projects, where many 
factors may affect the success or otherwise of an innovation. This kind of qualitative, evaluative 
research requires great skill, if it is not to degenerate into entirely subjective and unverifiable 
personal opinions, but when well done, it is often the most useful kind of institutional research. 

7. Creating a framework for research and evaluation 

Why has tfiere been so little research and evaluation? 

One important principle was established when the terms of reference were agreed. It would not be 
appropriate to set up a large, separate department or institute of institutional research within either 
organisation. It was agreed that research, evaluation and development should be the responsibility 
of gyery department. The creation of a large, separate research department was likely to lead to a 
separation of research from decision-making. In any case, given the size of the two organisations, 
the cornmitment of substantial internal resources for research and evaluation would not be feasible. 

However, if it is agreed that research and evaluation is the responsibility of every department, why 
has it not happened in the past? What is currently stopping staff from doing it now? 

Most of the staff had not until interviewed thought that it was part of their job to conduct 
institutional research activities. They assumed that these were not part of course production, and 
consequently 'the management' would not approve, indeed would actively disapprove, of such 
activities. (This was quite surprising, as nearly all senior managers interviewed said that they 
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thought it was good that staff should evaluate their activities, and said they would like to encourage 
people who wanted to do this.) Many staff also said that even if it was pait of their job role, there 
was not sufficient time to conduct such activities, given current production and maintenance loads. 
A lot of staff said that they would like to conduct reseL A and evaluation activities, but as well as 
lack of time, they did not have the expertise to do it. 

Quite clearly, an environment has grown up over the years in both institutions (more so in OLI than 
KN), in which research and evaluation activities have been discouraged. That may no longer be the 
management policy, especially since the recent re-organisation, but if such a change of policy 
exists, it needs to be communicated. However, telling staff they can now do research and 
evaluation is not enough. It is necessary to provide a supportive framework or environment which 
both encourages and at the same time controls and directs the development of research and 
evaluation activities. 

Staff are also unsure as to what steps need to be taken to evaluate systematically their activities. 
Some kind of guidance or structure is therefore necessary. 

The main purpose of research and evaluation should not be summative (i.e. to be used for 
promotion, external rewards or punishment) but formative (i.e. for people to be able to do their 
jobs better, through taking responsibility themselves for changes in practice). The principle 
therefore is self-evaluation, with help and guidance from senior management. The guidance should 
take the form of providing a structure, or series of events, which prompt self-evalua^on activities. 

Recommendation 

3. Each programme area or department should be responsible for evaluating its own 
performance; consequently, each programme area and department should be responsible for: 

(a) identifying and reviewing its own performance criteria, and having these 
accepted by the next senior level of management; 

(b) identifying the information it needs to collect on a regular basis in order to 
evaluate its own performance, and collecting and analysing that information 

(c) identifying and implementing special research projects which aim to enhance 
the performance of the department 



(d) formally reviewing, at least once a year, its own performance, in the form 
of a report to the next senior level of management; it is at this point that next 
year's performance criteria should also be considered and agreed. Research 
and evaluation data would be fine of the sources of information on which 
this review would be based. 

(e) department and programme area heads should provide a short report as a 
result of this review to the CEO, and these reports should be discussed at 
the Executive Council. The CEO should write annually a summary report on 
performance for the Board meeting (in May?), using both these reports and 
statistical information on course enrolments, completion rates, and trends 
(see recommendations regarding the data-base for more details). 

Project management and professional development 

If research and evaluation (not to mention development) are to take place, some way has to be 
found to free up time from day-to-day operational activities. There are two ways in which this can 
be done. 

The first is through the extension of project management to research activities. Thus a research 
study can be treated in exactly the same way as the new project management procedures for a 
course, with an approval procedure, a plan for resources, and staff time allocated to the project. 

The second is through an allocation to staff of a limited but fixed amount of professional 
development time, during which research and evaluation activities might be carried out. Of all the 
proposals floated in the first round of discussions, this has probably received the most attention, 
and has been the most misunderstood, so it is necessary to explain both the reasons why 
professional development is so important, and exactly what it covers. 

Good research and development needs to be encouraged from the "bottom-up", i.e. from people 
working close to problem areas, who know the problems well, who have knowledge of new 
developments in their field outside of their own institution, and who have ideas about how to get 
the information needed, or provide solutions to problems. This means encouraging staff to take the 
initiative in proposing and conducting evaluation, research and development activities. 




The importance of this cannot be too strongly stressed if OLI and KN are to become centres of 
excellence, and the OLA a world-leading agency in open and distance learning. I was disturbed by 
the restricted job role otherwise dedicated and committed staff ascribed to themselves. Many 
complained that in the past, any attempt to innovate, research or evaluate had been slapped down 
and discouraged by 'the management 1 . It was claimed that good staff who had wanted to develop 
their professional skills through private study had been discouraged and left. Whether or not this is 
true is not important; what matters is that many staff JbsdlCYC it to be true. Many staff feel extremely 
frustrated and under-utilised as a result. 

The consequence has been what I can only describe as a civil-service attitude to work. Initiative is 
discouraged, and people become driven by the production line. It seems therefore that the OLA 
Board needs to address a major policy decision. Are the two organisations to become leading 
innovative institutions driven by internally determined criteria of high educational quality, research 
and development, or are they to be administrative arms of the provincial civil service, driven by 
rules, regulations and external criteria? If the former, the staff must have conditions of service 
which encourage research and development, and the most important of these is the opportunity for 
professional development. 

It needs to be understood that the argument for professional development rests not on the benefits 
to the individual (although there is no denying that it will bring such benefits), but to the institution. 
Both OLI and KN are 'high-tech* institutions. Any institution working in high technology areas 
will die without professional development, as experience in the commercial sector indicates. 15% 
of IBM's turnover goes towards professional training and development. The Director of IBM 
Education, Europe, who has a budget of $1 billion a year for professional development within the 
company, claims: "We are big because we train; we don't train because we're big." Indeed, there is 
an irony that an institution such as OLI, which is trying to provide professional development and 
training opportunities for students working in other organisations, does not have a planned 
professional development programme for its own staff 

Professional development is too important to be left to ad hoc arrangements. It needs to be planned 
for, and organised in such a way that regular production activities are either covered or reduced. 
One objection to specifying a definite amount of professional development time was that there is no 
cover for certain staff. It is not good management policy though to make anyone totally 
indispensable. People become sick, leave or die on the job. Planned professional development 
leads to better back-up and cover, because people have to be trained to take over other jobs while 
covering for people away for professional development purposes. 




If professional development is not planned, the pressure of day-to-day operations will override the 
necessity for training and development The good intentions of managers - and dedicated staff 
themselves - to allow for professional development quickly disappear under operational conditions, 
particularly when organisations are continually being asked to do more with less resource. For this 
reason, there needs to be a strong countervailing pressure on managers to plan for professional 
development That countervailing pressure needs to come from the staff themselves. Consequently, 
a minimum amount of time for professional development needs to be negotiated and agreed with 
staff, so that other activities have to be planned around the time remaining. Without professional 
development time being a right; for the staff (albeit one which needs to be exercised flexibly), 
professional development is unlikely to happen. 

This is not quite such a radical suggestion as some have believed. There is already some 
professional development going on (attendance at conferences, overseas consultancy, visits to other 
institutions) but it is almost unofficial, restricted mainly to senior staff, and certainly not planned or 
seen as a right Institutional research would be only one, relatively minor, professional 
development activity, for most people. However, time for professional development will provide 
staff with the opportunity to follow through research issues of relevance to the institutions, but with 
the initiative coming from the staff themselves. It will legitimise, for instance, the activities of 
several committed staff who currently feel guilty if they spend an afternoon trying to find out why 
so many students are not completing a certain course, instead of getting on with new course 
production. 

Lastly, the suggestion that professional development time should be a right for staff does not mean 
that staff can do what they like with that time. The activities have to be negotiated with managers, 
and they must occur at times that do not disrupt unnecessarily other operational activities. Within a 
period of 12 months, though, a member of staff should be able to claim the full amount of 
negotiated professional development time. 

A final objection to this proposal is that it will be seen as an extra activity, will have to be costed as 
such, and will thus fall foul of the provincial government's policy of restraint. This is not 
necessarily so. There are really three alternatives. 

The first is not to accept the proposal, with all the risks that that entails. 

The second is to reduce proportionately production activities. This would not be disastrous, 
because if the reduction in time is converted directly into reduction in the number of courses, the 
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reduction would still be relatively small (a maximum of 10%). However, it is unlikely that even in 
this extreme case, efficiency would suffer. A reduction in incompletion rates by 10% as a result of 
research and professional development would maintain the throughput of qualified students at the 
same rate as at present, even with fewer courses. In practice, increased course production 
efficiency (due to greater knowledge and commitment) is likely to more than compensate for a 
reduction in time spent on course production, enabling current production levels to be maintained. 

The third alternative is to cost professional development as Jin extra activity, and to ask for more 
resources. This does not seem to me to be a sensible policy. The whole aim of professional 
development is to make the system more effective, not more expensive. Nor does it look good to 
admit publicly that there has been no professional development in the two institutions over the last 
eight years. What is really being suggested is a re-organisation and better planning of activities. 

Recommendations 

4. The OLA Board needs to disseminate to staff of both institutions the fundamental change in 
stance regarding their role following the adoption of the Mission and Goals statement; the 
implications for working style and professional development need to be explained and 
understood by staff at all levels, but especially by senior managers, who need to take positive 
steps to encourage professional development. Without this, the climate necessary for good 
quality research and evaluation (and, more importantly, for achieving the broader Agency 
goals) will not exist. 

5. Every member of staff should be encouraged to have a planned set of professional 
development activities over a year's period. These activities should be of benefit both to the 
individual and the Agency. It should be the responsibility of each manager to negotiate 
professional development activities with staff in their department. Each person should be 
entitled to the equivalent of half a day per week, or 25 days per year, minimum, for approved 
professional development activities, to include personal research activities, attendance at 
conferences, training courses, visits or exchanges with other relevant institutions, overseas 
consultancies, approved courses of study, work on new developments, and other 
non-operational activities which lead to increased competence in the post. Thus the 25 days 
will include a number of activities already being carried out. This general principle should 
apply to ail levels and grades of staff, although the amount of time for professional 
development activities may vary across grades. 
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I have suggested a particular mechanism for ensuring that professional development takes place: 
that staff should have a right to negotiate a minimum of 25 days professional development time. 
There arc other, perhaps better mechanisms, which might be introduced by management to provide 
professional development. I wish to stress that the issue here is not the particular mechanism, but 
that: 

(a) relevant professional development does take place at all levels within the 
organisations 

(b) whatever mechanism is adopted, it should encourage the generation of research and 
development activities by all levels of staff 

(c) it is a management responsibility to ensure that professional development happens at 
all levels of staffing. 

The need to improve contact between the students and central staff 

All distance education systems suffer from the gap between students, on the one hand, and central 
staff, both managers and course designers, on the other. OLI, and to a lesser extent KN, do 
provide excellent opportunities for contact between tutors and central staff, through workshops and 
one-day conferences. These are extremely important, because tutors are an invaluable source of 
feedback. However, there are considerable advantages in increasing direct contact also between 
students and central staff, despite the genuine difficulties, because student perceptions arc different. 

Part of the function of research is to collect feedback from students. This can be done through 
questionnaires, telephone interviews and a number of other techniques. However, they all have 
their limitations. In particular, it is extremely difficult to obtain well-considered, deeply insightful 
comir^nts from students using these more formal research techniques, and in particular, it is 
difficult to deal with arguments, negotiations or discussion of issues through formal research 
methods. Student criticisms also tend to lose their force when aggregated or presented indirectly. 

It is important therefore not to overlook more direct methods of student contact and involvement 
with the centre. This is always going to be difficult, because students are not only often a long 
distance away from the centre, but they have busy lives, working and studying, and many have no 
wish or interest to become involved in the working of the two institutions. The few that do become 
involved then are not necessarily representative of the whole student body. 
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However, that does not mean that one should give up trying. There are students who are interested 
and willing to participate directly. For instance, several OLI staff are also (or have been) students. 
Over 30% of the students do live in the Vancouver area, and some will have the time and interest to 
get involved in the work of the two institutions. 

Recommendations 

6. There are several ways in which central staff can be brought into meaningful and useful 
contact with students and tutors: 

(a) appointment of a student and a tutor representative to the OLA Board 

(b) the appointment of a paid student editor for the OLA newspaper, with a 
remit to increase the number of student subscribers and contributors 

(c) appointment of a student or tutor to selected project management teams 

(d) establishment of a students 1 association, with elected student 
representatives; OLA to provide audio and/or computer conferencing, 
including the use of a students' bulletin board for elected representatives 

(e) provision of two or three self-operated video boxes, which record viewer's 
comments, and which can be broadcast back to the audience at a later date, 
to provide direct feedback on KN prograiunes 

(f) occasional weekend student/tutor get-together days, within programme 
areas, to discuss general student issues (study techniques, course choice), 
etc., on an optional, voluntary basis. 

The need for both regular and special research and evaluation activities 

Because of limited resources, not just for research but also for re-design and re-production, it is not 
feasible to research every course in depth. There is no point in collecting evaluation data on a 
course if it cannot be acted on. Furthermore, there are important areas which support course 
production and delivery which may also need special studies. It therefore becomes essential to 
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concentrate evaluation activities in those areas where they are most needed. On the other hand, 
every course should be assessed to some extent, since it would be unsatisfactory to run a course 
which is inadequate. Therefore two quite different kinds of research and evaluation activity are 
necessary: a minimal but adequate process of course evaluation for all courses; and a limited 
number of special studies, in the form of research projects, which may be directed at a wide range 
of possible issues, but would also include special in-depth evaluations of selected problem courses. 

Recommendation 

7. Two kinds of research and evaluation activities for the Authority are proposed: 

(a) regular activities (such as course evaluation) which are built into project 
management; 

(b) special projects: individual studies, commissioned or approved either within 
KN or OLI, or in conjuction with other agencies 

In either case, procedures are proposed to ensure that each type of activity is approved. 
Toperly resourced and professionally conducted (see sections 9 and 10). 

8. Using the OLI data-base for research and evaluation purposes 

OLI has an excellent store of information about students, tutors and courses. This information is 
invaluable for research and evaluation purposes. Many useful studies can be conducted (for 
instance, on factors associated with non-completion, the study routes chosen by students) without 
the need for collecting additional information. 

Howev:* , a major complaint from staff is that the data-base, as currently structured, is useless for 
research or evaluation purposes. The problem is that the data-base is not organised at the moment in 
such a way that basic information about courses or students required for evaluation purposes can be 
readily abstracted. Much of the information required can be obtained through using a software 
package called Datatrieve, but it requires extremely laborious procedures and a great deal of 
knowledge, both of the data-base and the Datatrieve programmes. Other information is in fact 
impossible io abstract in the form necessary. This limits information about basic performance 
indicators to a very small number of people, all of whom have other, more pressing operational 
priorities. Unless this bottleneck is removed, evaluation of courses on a systematic and reliable 
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As an example, I examined what was needed to conduct an enquiry from Jocelyn Calvert regarding 
drop-outs (see Appendix D). After discussion with Heather Drugge, Student Services, who 
frequently uses Datatrieve, it became apparent that it would take 40 hours of Heather's time to get 
this information. Despite the fact that I am an experienced researcher, and familiar with data-bases, 
it would have taken me considerably longer. This is not practical, yet the enquiry is a reasonable 
one, and the information required for the enquiry is actually in the data-base. 

The problem is caused because individual student files are not complete. There are three file 
systems. The master student file is what is entered once only from the initial registration form. 
Then there is a master course file for each course, which lists details of the course, but nQL student 
information. A third file is then created (the student course file) for an individual course, which 
takes information from the master course file, 'reads' the student files, then identifies which course 
the student is registering for, then transfers the student numt>er only to the student course file. This 
leaves important demographic information in the student master file. To get any other information 
about students on any course requires special programmes to be written, or the use of Datatrieve. In 
other words, to match student performance with other student information, a special record has to 
be created for each student each time. 

What is required is a programme which automatically merges new student data with existing 
student data, in a consistent way for all students. Thus after a student registers for a course, not 
only should course performance information (e.g. grades) be added to the master student course 
file, it should also automatically be added to the master student file. Similarly, when a student 
registers for a second (or subsequent) course, all the student information regarding the second 
course should be added to the initial master student file. All fields on the file should be keyed 
fields. Thus one is creating an easily accessible, dynamic, continually up-dated and above all 
complete record for each student. Once such a file is created for all students, this becomes the 
master data-base from which all subsequent analyses can be done for evaluation and research 
purposes. In other words, the computer will search through the master student file according to 
parameters set by the enquirer - for instance "Select all students on courses with ADMN codes for 
all sessions, who previously studied an OLA pre-university course". This would then provide the 
sample on which further analyses can be done. Using this system, it should also be feasible to 
abstract appropriate sample information for local processing on micros, thus reducing central 
processing time. 
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While this is a requirement for evaluation and research purposes, it will also become increasingly 
necessary as more and more students complete degrees, or seek credit transfer, since a full student 
record will be required. Also, some countries now require, under public data protection acts, that 
all stored information about an individual should be available to that individual on request. My 
proposal would make this a much simpler process. 

Once this basic master student file is created, standard statistical analysis packets (such as SPSS) 
can be bought, which will enable data to be easily abstracted in the form required by anyone 
authorised to do so, whether or not they have programming skills. 

Information Systems will need to do an initiation study to see what resources are required to 
provide a basic, dynamic master student file, and what packages would be most appropriate for 
abstracting and analysing data, including standard statistical tests. 

It is important to point out that re-structuring the data-base is essential for project research. The 
present system makes it impractical to sample students, send questionnaires, or conduct analyses of 
student demographics, on any regular or systematic basis. 

. The proposals to be recommended would also have value on a wider agency basis. A common 
identifier for all B.C. students in post-secondary education is being used. The data-base changes 
would enable students to be traced across several different institutions, and where appropriate, 
additional data from their time at other institutions can be added to the student master file. 

* These suggestions have been discussed with Information Systems staff, who agree the proposals 
are feasible. It is a question of setting priorities for implementing these proposals. 

Recommendation 

8,1 Improvements to the data-base along the lines proposed should receive the highest priority in 
the Information Systems development programme. 

Detailed proposals follow on what needs to be ''one, but it is also important that general principles 
are established regarding the use of the data-oase for research and evaluation purposes, so that 
information accessed is directly relevant to, and used for, decision-making, and to ensure that time 
of both people and the central processing unit is not wasted. 

*5 
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Recommendations 

8.2 Data from the data base should be accessed for a specific reason, related to policy or 
decision-making; this means there should be specific times when the information is needed, 
and a specific person or set of persons who would be committed to using this information. 

8.3 Routine or regular provision of information should therefore be kept to a minimum, and 
related to particular regular decisions or policy requirements. 

8.4 Self-access should be the main way in which the data-base should be used. In other words, 
instructional designers, middle managers, and senior managers (or their assistants) need to 
be able to get information from the data-base relatively easily. Those people who need 
information should not only be responsible for defining their needs, but should wherever 
possible access it themselves, at times when they need it, and without having to learn 
specialist programming skills; advice though should be available on the best means to obtain 
the information required. Appendix D outlines some of the analyses which it ought to be 
possible to access easily and conveniently. 

8.5 It is important to define the research and evaluation needs of different groups of people and 
to develop procedures of data abstraction which meets those needs. The following needs 
have been identified: 

i Course maintenance. It will be recommended (9.5) that each course should be 
reviewed as soon as possible after the first semester presentation of that course. 
Thus for each new course, the print-out in Appendix F should be commissioned by 
whoever is responsible for course maintenance. (A programme for this already 
exists). If any further information is needed from the data-base, this should be 
acquired by the maintenance person by directly accessing the data-base. 

ii M iddle management (directors, course co-ordinators, possibly vice-presidents). 
They need an overall picture of the progress of courses m their areas. It is 
recommended that this is necessary only once a year , rather than semester-by- 
semester, as at present. Once a year, middle management should get the print-out 
outlined in Appendix G, covering a full year period, i.e. for all students registered 
each semester within a specified year-period. All course maintenance people should 
also automatically get a copy, so they can see how their individual courses compare 
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iii Executive Council and the Roard The Executive and the Board need an annual 
picture both as a form of accountability, and to help inform discussions regarding 
future policy for the Institute/Authority. A statistical summary (as outlined in 
Appendix H) therefore should be provided to the Board and Executive Council. The 
annual course statistics should cover a full financial year, to make it easier to relate 
expenditure to performance. This summary should be the main document, but 
Appendix G would also be available to the Executive Council and Board members, 
for those who want to look more closely at the figures. 

8.6 Where it is not possible to access easily the required information, or where a detailed or 
complex investigation is required, this should be treated as a project, with the consequent 
need for approval and allocation of suitable resources, and setting of priorities. These 
enquiries would entail the use of a specialist to obtain* the necessary information, and 
possibly special computer programmes to be written. 

8.7 The existing tools for or ganising and extracting data are not adequate for allowing easy and 
convenient self-access for the purposes of undertaking statistical analyses. What is required 
is a programme which automatically merges new student data with existing student data, in a 
consistent way for all students, to create a dynamic, up-to-date master student file. 

8.8 Once this basic master student file is created, standard statistical analysis software packages 
(such as SPSS) can be bought, which will enable data to be easily abstracted in the form 
required by anyone authorised to do so, whether or not they have programming skills. It is 
recommended that such a package capable of running on the mainframe should be purchased. 

8.9 For research or evaluation enquiries requiring substantial statistical analysis or cross-breaks, 
the appropriate information required (e.g. information about students on a particular course) 
should be abstracted from the master data-base for local processing on micros, thus reducing 
CPU time. This would require purchasing one suitable micro for each main programme area 
(i.e. approximately five) plus suitable micro software packages, to provide statistical 
analyses and cross-breaks. Thus simple analyses could be run directly on the mainframe; 
more complex analyses would be done on micros, to reduce CPU time. 

8.10 In order to provide accurate costing and an efficient implementation of these 
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recommendations, Information Systems should be requested to do an initiation study to see 
what is required to provide a basic, dynamic master student file, and what packages would 
be most appropriate for abstracting and analysing data, including standard statistical tests. 

9. Course evaluation 

A two-tier approach to course evaluation is proposed, with a standard, minimal evaluation 
procedure for all courses, and a special evaluation procedure, or project, for selected courses only. 

Changing course material 

It is important to distinguish between two quite different kinds of reason for making changes to a 
course. The first is because the course is not viable as it stands. This may be because the text book- 
on which it was based is no longer in print, because there are fundamental flaws in the teaching, or 
because the assignments bear no relationship to what is being taught. In these cases, changes are 
absolutely necessary, resources must be found, or the course must be withdrawn. 

The second eason for making changes is because although the course is quite viable as it stands, it 
could be improved. It is recommended though that changes should be limited in this case to one 
main batch which can be made at very low cost, early on in the life of the course. Resources for 
remaking courses are limited, and should be concentrated on those courses most in need of 
changes. 

The goal then should be to aim for high quality courses robust enough to last without the need for 
constant changes. There is a set of procedures thai will usually allow course designers to reach that 
standard, and OLI course designers by and large are aware of those procedures. 

Changes to courses after first presentation divert resources away from initial design. In the end, 
constantly changing courses when resources are limited means that either poor quality courses are 
offered initially, because insufficient resource is available for course design, or people who could 
be working on new courses are still revising existing courses. That in the end means that students 
suffer, because they get less choice. 

Standard course evaluations 

The standard method of course evaluation being proposed is dependent on the concept of project 
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management. It is also dependent on a relatively small amount of resource being set aside for each 
course to allow for changes. Lastly, it is dependent on a properly planned and resourced procedure 
for looking after a course once students are enrolled (i.e. course maintenance). Evaluation should 
be part of that course maintenance process. 

The current lack of consideration given to the presentation of a course after it has been handed over 
for production is a weakness. This is when the learning begins, not ends. If project management 
means anything, it has to be responsible not only for the design of the course but also its conduct. 
Project management is likely to be in the hands of one person during maintenance, but at the 
planning stage, decisions need to be taken about how the course will be managed during 
presentation. Evaluation, and the changes that may need to be made as a result, should be a pan of 
that maintenance activity. 

Special evaluations 

Special evaluations, because of the cost, should be used selectively. Some reasons which might 
justify a special evaluation would be: 

(a) external funding for evaluation, or because course sponsors or accreditation bodies 
demand evaluation 

(b) a new venture: new target group, a new programme area, or a radically new course 
design 

(c) a high profile course Qarge numbers of students, government or public interest, etc.) 

(d) unusually poor performance of a course or programme, as identified by standard 
evaluations, without reasons being clear 

Special evaluations should be treated as projects, with resources requested not only for the conduct 
of the evaluation, but also for the changes that will follow as a result. It would be difficult to justify 
a special evaluation if it was not intended to make substantial changes. 



An argument against piloting 



In some areas, piloting of courses is becoming popular, that is, trying out materials on the first 
semester students, then revising materials subsequently. This is being done to take advantage of 
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desk-top publishing, which enables changes to print material to be made quickly and easily, 
without typesetting. There arc several serious objections to the way piloting is being conducted in 
OU at the moment: 

(a) it is increasing the cost of course design and in fact slowing down the rate at which 
new courses are being produced 

(b) it is causing major problems in course delivery and student services 

(c) it raises questions about the validity of a credit when different students are studying 
different versions of the same course 

(d) there is an ethical problem of 'early* students receiving a poorer version than later 
students 

(e) early versions with a large number of errors and poor design create a bad image for 
the Institute 

(H changes are not usually made on the basis of system atic feedback and evaluation 

It is accepted that In certair areas, it will be difficult always to get the right approach to new types 
of students or new subject aieas first lane; also, it will be necessary to change courses from time to 
time because of significant changes in subject matter (new regulations, unavailability of text 
books). 

However, there are better ways to handle these problems than deliberately designing material in 
rough pilot forms which are studied by fully enrolled students, then continually changed. Where 
doubts exist about the target group, this should be handled by a careful study of the target group 
before course design begins, i.e. as a market research project. If a course still turns out to be 
unsuitable for the target audience, it is recommended that this be handled through the procedure for 
special evaluations. If significant changes in the subject matter during the life of a course require 
changes to the course maerial, this should be handled as part of project management. In other 
words, maintenance resources need to be reserved for unexpected changes, or where it is known 
that changes are likely during the course, course design should allow for changes to be made 
easily, through the use of separately printed supplementary material for those areas likely to 
change. 
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Recommendations 

9. 1 Course evaluation should be considered as a component of the project management process 
for course design and delivery, for every new course. 

9*2 Courses should be designed with the firm intention of running for at least six semesters 
without changes; in most cases, changes after that date would be restricted, and limited 
wherever possible to one set at a date and within resources agreed in the planning stage; in 
other words, the aim is to design courses which are viable from the first date of presentation 

9.3 The following course evaluation decisions need to be made at the project management 
planning stage: 

(a) discussion of overall evaluation strategy, followed by a decision regarding whether a 
standard or special evaluation approach should be adopted for the course, or whether 
market research is needed to identify better the requirements of the target group 

(b) evaluation costs, work-load and responsibilities, and the timing of the evaluation 

(c) the implication for resources, production, delivery and tutoring of any changes likely 
as a result of evaluation, or changes to course content during the life of the course. 

9.4 A standard evaluation approach would ensure that every new course is reviewed at a set time 
soon after first presentation. Details of how this is done can be refined and possibly 
simplified, but one suggested procedure is as follows: 



(a) every new course should be reviewed within six months of the first semester of 
presentation (or as near as possible to that time, if examination arrangements prevent 
results being available within that period) 

(b) collection and analysis of information for the course review should be seen as pan of 
the maintenance of a course (i.e. the management of a course during its 
presentation); one person (i.e. the person responsible for course maintenance) 
should be responsible for co-ordinating evaluation information. 

(c) an evaluation file for each course should be opened from the start of course design : 
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into this file would go course developer's doubts and queries about design, course 
consultant's comments, delivery and tutoring issues, errors spotted after delivery, 
letters and phone-calls from students, advisers and tutors, and any emergency action 
taken; the file should be kept open for the whole life of a course. 

(d) emergency action would be taken within the first six months of course presentation 
only in exceptional cases (i.e. where serious consequences likely to lead to 
widespread student failure would result because of not acting). 

(e) the person responsible for course maintenance should commission within six 
months of the first presentation of the course: 

i a standard print-out of the first semester enrolments and grade distributions 



iii. a statistical analysis of assignment and examination questions as soon as 
they are available; these can be compared with data from other comparable 
courses; if necessary, more data (e.g. student demo graphics) can be called 
up from the data-base (recommended changes to the data-base will make 
this procedure much simpler and easier than at present). This information 
would be added to the evaluation file. 

(f) The senior tutor or course co-ordinator should formally contact tutors (and exam, 
markers if different) by phone after the first examination for detailed comments on 
the course, using a semi-structured set of questions, designed in conjunction with 
the course developers. The senior tutor or course co-ordinator would then write a 
report on the course based on this tutor feedback. This report would be added to the 
evaluation file. 

(g) the senior tutor or course co-ordinator would then prepare a report on the course, 
using all the information in the evaluation file, including a recommended course of 
action. A copy would be sent to the head of the programme area. 

(h) if the course is considered to be generally satisfactory, or changes can be 



(see recommendation 8.6.i.) 
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tutor statistics (see recommendation 12.3) 
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accommodated within the maintenance budget, the evaluation file would be kept 
open by the course maintenance person, who would also be responsible for drawing 
attention to any subsequent developments which might require unplanned changes to 
the course. 

(j) if there arc major problems, which cannot be easily explained from the standard 

evaluation (e.g. an unusually high drop-out rate), the head of the programme area * 
would decide whether to bid for funds for a special evaluation; whether to leave the 
course as it is; or whether to withdraw the course altogether. 

(k) where changes are being proposed to an existing course, tutors should be consulted 
about the proposed changes. 

these recommendations need not be followed in detail, but provide an example of how a 
standard evaluation procedure might be conducted. 

A special evaluation of a new course may on occasion be appropriate, but as an exception 
rather than a general rule. This would involve a more detailed study of a course, and would 
usually be linked to a prior commitment to substantial changes to the course as a result of the 
evaluation, if this proves necessary. Special evaluations, because of the cost, should be used 
selectively. One procedure would be as follows: 

(a) If a special evaluation is decided at the initial planning stage, it should be timed so 
that information is collected during the first and possibly second semester; 
otherwise, information should be collected as soon as possible after a standard 
evaluation has identified problems. It will usually be necessary then to run a course 
for several semesters before a revised version can be introduced. 

(b) Special evaluations should use postal questionnaires and/or telephone interviews for 
students, and any informal group discussions that can be arranged, as well as the 
procedures outlined for standard evaluations. 

» 

(c) Once the information is collected, decisions should be made about necessary 
changes, within the budget allocated; in addition, an executive summary should be 
widely circulated and an open seminar held to discuss the implications of the report. 

Piloting or pre-testing of courses should be treated as a special evaluation. 

ro 
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9.7 It is recommended that each major programme area should review its activities annually. This 
would require a meeting of the staff within a programme area, chaired by the head of the 
programme, at which courses, delivery, tuition and other issues would be discussed. The 
aim of the review would be to identify necessary changes to next year's programme of 
activities, and might coincide with budget projections for the following year. For this 
meeting, a paper should be drawn up which provides a summary of enrolment and grade 
information for all courses in the programme area over the year period (see recommendation 
8.6.U.). Course evaluation reports prepared during the year would support this 
documentation. 

10. Project research 

There are two different kinds of project to be considered. The first are development projec t- which 
may or may not have a research or evaluation component, and the the second are research projects , 
i.e. projects which are fundamentally research enquiries, but not necessarily with a development 
aspect. 

Projects arc considered to be well-defined activities of initially limiied duration. In o(i:vr words, 
resources are committed for a definitely limited period, although if, in the case of a development 
project, it is considered successful, the activity may convert into an on-going operational activity. It 
would then though no longer be a development project. 

Development projects 

There are very strong arguments for ensuring that research and evaluation are built into 
development projects. Research and evaluation can: 

(a) improve the project during its operation 

(b) provide external credibility 

(c) account for the way resources have been used 

(d) enable lessons to be learned which may apply to other projects/activities 
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(e) help decide whether or not to operationalise/continue with the development activity 



A development project may have one or more of the following criteria for success: 

(a) improvement in quantity (doing the same as before, but more 
cheaply; or getting more for the same money) 

(b) ' ™improvemenrmiquaU^^ 

(c) increasing activities (doing something new: reaching new audiences, teaching a new 
subject, bringing in additional resources) 

By their nature, development projects tend to be exploratory. Often then the benefits or limitations 
of the development will not become apparent until some way through the project, and may be quite 
unanticipated. Also, whether or not a development succeeds is likely to depend as much on the 
conditions under which the activity operates as on the technology or development itself. For these 
reasons, research and evaluation methodology, and criteria forjudging success, need to be flexible. 

Lastly, it is assumed that development projects will be assessed, approved and managed in the 
same way as other projects. 

Recommendations 

1 0. 1 Evaluation should be built into project management. This means: 

(a) discussing the evaluation strategy at an early project planning stage 

(b) determining at this stage the necessary resources (manpower, funds, and timing ol 
evaluation) 

(c) conducting evaluation as a defined, continuous activity throughout the project 



(d) defining criteria for success. 



c 



10.2 Evaluation designs for development projects should be flexible, to reflect the context of the 
project, and the often exploratory nature of development projects; initial objectives for 
instance may legitimately be revised as a result of experience. 
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10.3 It is important in the evaluation of development projects to look at process as well as 
outcomes (how best to do things, as well as looking at costs and learning results). This may 
require: 

(a) a qualitative, observational approach 



-(b)-— identification and recording of key events in^projecr ™ ~~~~ 

(c) explanation of why events/decisions were made, as well as description 

(d) examination of the teaching or decision-making process, examining, for instance, 
the nature of interaction between learners and teachers, or what kinds of learning 
take place as a result of developmental activities 

(e) analysis of the development's impact on work roles, need for new skills, training, 
etc. 



(f) resource implications, not only in terms of costs incurred, but how a new 
development affects the balance of costs between different activities and spending 
departments (e.g. audio conferencing may require a shift of costs from production to 
delivery) 



10.4 For these reasons, it is recommended that evaluation of development projects should 
generally be conducted internally, i.e. by project staff, but with external assistance from a 
professional evaluator, acting as a consultant, wherever possible. This means that evaluation 
resources may better be spent increasing the internal manpower of a project, to allow internal 
staff to have adequate time for evaluation activities. Credibility will depend then on high 
quality, clearly written evaluation reports, which allow those not directly involved to 
understand the development project, its achievements and difficulties. 

10.5 To ensure high quality evaluation, it is recommended that evaluation should account for a 
minimum of 10% of internal manpower resources on a development project. 
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Research Projects 



A large number of research and evaluation projects have been suggested by staff at OLI and KN. 
Good research requires careful selection of topics and methods, commitment, skill and interest 
from those conducting the research, and a decision-making framework that will facilitate changes as 
a result of the research. 

'Research projects into open learning and distance education may originate from the following 
sources: 

(a) The OLA Executive Council may wish to research significant areas of concern to 
assist decision-making. 

(b) Programme areas and operational units , who will wish to conduct research related to 
courses, problems or developments specific to their areas. 

(c) Individual staff - who wish to pursue a particular research topic associated with their 
area of work. 

(d) Externally funded or initiated studies, concerned with general research problems hi 
open learning and distance education, but requiring use of or access to Open 
Learning Authority data or students, or needing to work in conjunction with 
OLI/KN. 

Research projects then will have a number of different clients, so it is advisable not to centralise all 
resources for research projects. At the same time, it is necessary to have formal approval 
procedures to ensure that worthwhile projects are properly condu*. *.ed. 

In setting priorities, it is important to have clear criteria. I offer the following as guidelines for 
research project approval: 

1. Is the research likely to increase the effectiveness of open learning in British 
Columbia? 

Is the research likely to enhance the reputation of open learning in B.C.? (Note that 
this does not mean that potentially critical studies should not be undertaken; it is 



2. 
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equally important to demonstrate that activities are being carefully analysed and 
assessed) 



Is the research likely to identify major problems in open learning which need to be 
identified and addressed in B.C.? 

Is there a clear client for the research, and is the client likely to act on the findings of 
the research? 



Is the project likely to be competently conducted, with appropriate methodology and 
timing? 

Is the project likely to give a good return for the resources invested? (Small projects 
with a limited goal are often more likely to succeed than large projects with 
grandiose aims.) 



Lastly, it is assumed that research projects will be assessed, approved and managed in the same 
way as other projects. 



Recommendations 



11.1 All research projects should go through a formal approval procedure, to ensure projects of 
significance are undertaken, to avoid duplication and to ensure that they are competently 
conducted. 

(a) Bids can be proposed by any individual or group of staff, but should be discussed 
with their superior manager and referred upward for approval. 
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(b) The Executive Council should assess and if appropriate approve any single research 
proposal requiring more than $50,000 of resources (manpower and/or cash), or 
involving co-operation with external institutions, or paid for out of the Executive's 
own research and development budget. 

(b) Vice-Presidents should assess and if appropriate approve any project within their 
area requiring less than $50,000 worth of resources from within that area. 
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(c) The Director, Research and Evaluation Methods, should be consulted regarding the 
assessment of all research proposals (see recommendation 14.2) 



11.2 Research projects may be resourced from the following sources (see recommendations 
14.5-14.10): 

(*) rc-a^signmeifit of duties of internal staff (e.g. reduction in production activities). 

(b) a sum of money held by Executive Council (recommended amount: $100,000) 
and/or a sum of money available to each of the four component areas (recommended 
amount: $50,000 each); these sums of money may be used either to buy out internal 
staff from operational duties (using the money for staff replacements), or to hire in 
external research staff. 

(c) use of professional development time of internal staff. 

(d) external funding or assistance from other institutions. 

(e) any combination of the above. 

The aim is to provide flexible but limited sources of funding for research projects. 

11.3 In general, it is better to encourage staff to conduct research as a team rather than in isolation. 
Resources should be concentrated on a limited number of significant projects rather than lots 
of small projects. 

11.4 The approval procedure and sources of funding for development research projects 
should be the same. 

1 1 .5 Sources of external funding, and co-operation with external agencies, should be developed 
wherever possible. University post-graduate students should be encouraged to work with the 
OLI and KN, under joint supervision, to collect data and conduct research studies. OLI and 
KN should help University staff define significant research areas suitable for study by 
doctoral students. Joint studies on common areas of interest (e.g. drop-outs, learning at a 
distance, tracer studies following the progress of students) should be carried out jointly with 
other open learning and distance education institutio is, through both the proposed Centre for 
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Research and Education in Open Learning, and through the various Canadian distance 
education associations (CADE, WCCUDE). 



1 1 .6 However, the opportunities for such joint and external activities should not deflect OLI and 
KN from their responsibility to set their own priorities for research and evaluation, and for 
conducting their own internal research and evaluation activities. 

TTTTFis recommende31hat tfie following researcfTprojects should receive top priority. I have put 
them in my own order of priority, but the actual ordering of priority should be determined by 
the Executive Council: 

i Information Systems to develop a basic, dynamic master student file, and purchase 
of packages appropriate for abstracting and analysing data, including standard 
statistical tests. This should be a top priority within Information Systems (existing) 
development budget, since without it, evaluation and research cannot really begin. 

ii development of an accurate and practical means of costing OLA activities (essential 
for project management) - funded from (new) Administration research funds. 

iii Knowledge Network audience research study (see proposal 12.2): from (new) KN 
research funds, perhaps linked to (v) and (vi) below. 

iv at least one special course evaluation per year (as well as standard course evaluations 
on all other new courses) in each of the Open University, Open College, and KN 
programme areas (research funds from appropriate areas 1 research budgets). 

v profiles of the OLI student: a statistical analysis of age, occupation, education, 
motivation by programme area, and comparison with the general population of B.C.: 
using new data-base and new project officer (no other cost). 

vi market survey, to identify the public's knowledge and opinion of OLI, KN and 
OLA, and their perceived relevance to individual's needs: (new) Executive Council 
research fund. 

vii study of means to link OLA better to provincial and federal industry, commerce, 
government and voluntary organisations, to identify better their continuing and 




distance education needs, and to educate them into the potential of open learning 
(e.g. directory of agencies and contacts, high-level conference, etc.): from Executive 
Council Research fund, with staff seconded from relevant programme areas (this 
should be a low-cost project, using existing resources as far as possible). 

identification of obstacles to qualification through open learning; barriers to the 
pyramid of opportunity: individual case-studies, showing the perspective from the 
individu&^di^^ " 

tracer studies: what study paths do OLA students follow; what happens to drop-outs; 
where do OLA students come from (in terms of previous educational activities)? 
where do they go to? - from Executive Council research fund. 

project to identify whether there is sufficent field support for students, and if not, 
what steps can be taken to improve student support, and in particular whether the 
role and recruitment of tutors needs to be changed, and what role other universities 
and colleges might play in providing field support to OLA students to increase 
student support: from Executive Council research fund or Administrative research 
fund. 

study of specific B.C. open learning needs and the needs of minority groups: how 
does open learning look from their perspective; what are their special needs; what 
can be done for them? Some prospective groups: 

- long-distance fishermen (Open College budget) 
BC people working overseas (University budget) 

- lumber industry (Open College budget) 

- mining industry (Open College budget) 

really remote students (Administration research budget) 

- native Indians (Open College/ABE research budget) 
handicapped students (Administration research budget) 

- unemployed students (Open College/ABE research budget) 
English as a second language (Open College budget) 

No doubt other minority groups could be identified. 




An analysis of the reliability and validity of assessment questions (see proposal 
12.8): from Admiiiistratioiy budget. 



The above project recommendations should not preclude other studies being carried out, if 
resources can be found for such studies, and they are approved by the relevant 
Vice-President. 



11. Tutors, advisory services, and validation of student assessment procedures 

This is the most difficult area for me in making proposals. First, there is a major difference between 
my own institution and OLI/KN in the way student services are organised. This leads to some 
reservations on my part about the general policy in OLI/KN of keeping regional services to a 
minimum. In my own institution, 20% of the total budget is spent on regional services. Despite 
more than half of this going on administration (regional offices, etc.), there is in the British Open 
University an elaborate system of correspondence, telephone and face-to-face tuition, extensive 
counselling services, and as much personal contact between students and tutors as the system can 
manage. 

Not all students need or want extra support, one reason why many students study through open 
learning is because they prefer to study a'.one. However, for other students, at least in the British 
Open University, close contact with tutors and other students is very important, and helps to keep 
many students in the system who would o herwise drop out. Other studies of OLI (see for instance, 
Keegan, 1982) have suggested that student support services do need to be improved. 

70% of all 167 OLI tutors live in the Greater Vancouver area. Less than 10% live in areas other 
than Greater Vancouver and Victoria, compared with over 40% of the students. Does this matter? 
Can such tutors fully understand the needs of more remote students? Would a closer match of 
tutors to students in geographical distribution lead to more frequent, and more informal, contacts 
between students and tutors? Could suitable tutors be found in these regions, and what would the 
implications be for current contracts of dropping some existing tutors? Should tutors 1 roles be 
broadened slightly to provide more general advice to students on courses? Would a more equally 
distribution of tutors around the province lead to better communications with local colleges? What 
would be the implications for woik-load, training and budgets in broadening roles? 

I am aware of the major differences in the numbers and geography of students in the two systems. 
Even so, I have still found myself asking the fundamental question as to whether the balance of 




resources in the OL/KN between production/delivery on the one hand, and student services on the 
other, is right. In other words, do students get enough support? To answer that question, the 
possible role of universities and colleges in providing support for OLA students would need to be 
examined, and extensive research activities into student needs would have to be undertaken. This 
would be justified only if there was a policy commitment to increase or transfer resources, and/or 
commitment from the Universities and colleges to co-operate, and/or Provincial Government 

commitment to look at the funding of Universities and Colleges to see if this kind of support can be 

encouraged, if the studies did indicate the need to increase student services. Nevertheless, this 
seems to be such an important policy issue that evaluative research in this area should perhaps 
receive at least as great a priority as in course production. 

Even within the context of the OLI/KN, a very wide and diffuse range of activities is covered 
within the area of student services: relationships with local colleges and business, advice and 
information on courses and inevitably careers, tutor training and effectiveness, the effectiveness of 
delivery systems, and the validity of student assessment procedures. This spread of areas of 
interest makes it difficult to concentrate limited evaluative research resources. 

Thirdly, I have found it more difficult to pin down clear lines of decision-making and 
responsibilities in this diffuse area (itself an interesting issue), but this is essential if research is to 
influence decision-making. 

For these reasons, my proposals for research and evaluation in these areas are more tentative. 
Evaluation of tutors 

Tutors are an essential part of open learning systems. Their role in providing feedback on courses 
has already been discussed. However, tutors themselves have a need for professional developmeni 
and trying. 

At the moment, information about tutors is mainly statistical, and is used for mainly summaiive 
evaluation purposes. The present system of tutor evaluation is unsuitable, being based on a 
complex and time-consuming print-out of quantitative data on tutor assignments which is not easy 
to analyse, and which is not really justified in terms of the relatively small numbers of tutors 
involved (although it may be necessary for payment purposes). More direct methods of evaluating 
tutor performance should be used. 
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More importantly, tutors themselves often work in isolation. They need formative evaluation, in 
order to improvctheir skills as distant tutors. 

I am impressed by Athabasca University's computerised system for monitoring student and tutor 
progress and performance (TRIX). This requires every tutor to have a terminal, and to be 
responsible for entering data about student grades and perfonnance directly. It does though provide 
a very user-friendly system, enabling both tutors and course co-ordinators to be able to see at a 
glance which assignments have been submitted and marked, which students are behind schedule, 
and a range of other important indicators of student and tutor performance. This is the kind of 
sy^m that OLI needs, but supplemented by more qualitative information. 

Recommendations 

1 2. 1 Tutors should be evaluated on 

(a) their efficiency in turning round assignments 

(b) the reliability of their marking of assignments 

(c) the quality of their communication with students (comments on 
assignments, advice and help to students) 

These three criteria should be of equal importance. 

1 2.2 Senior tutors should be responsible for evaluating tutors in their subject areas. Where there is 
no senior tutor in a subject area, it should be the responsibility of the course-cordinator, but 
this should be only a temporary arrangement until a senior tutor post is available. The need to 
evaluate tutors should be taken into account in senior tutor workloads. 

12.3 A senior tutor should normally request a copy of at least one assignment, preferably more, 
per tutor per semester from registry. If there is cause for concern, the senior tutor should 
then request to see a sample of the tutor's next batch of assignments. It may be necessary 
as a result of such an inquiry to re-grade assignments at the discretion of the senior tutor. 
Senior tutors should also contact each tutor on a regular basis by telephone or othei means. 
Unless a system like TRIX is installed, senior tutors sh.n.ld also be sent each semester a 
print-out of assignments submitted and marked by all tutors in the subject area. (A computer 
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routine already exists for such a print-out). This would indicate turn-round time and quantity 
of assignments marked by each tutor. In exceptional cases, senior tutors may also want to 
contact students directly by phone if they are concerned about a tutor's performance. The 
main purpose of monitoring tutor marking and commenting would be to help tutors improve 
performance, rather than to assess the need to terminate employment, although that may be 
necessary in extreme cases. 



12.4 There should be an agreed and clearly publicised student appeal procedure regarding 
assignments. The senior tutors should deal with student appeals regarding assessment of 
individual assignments. The procedure, including to whom appeals should be made, should 
be printed in the OLI calendar. 

12.5 Senior tutors, in association with their course co-ordinators, should be responsible for 
organising occasional workshops for tutors, on assessment strategies, counselling, and 
giving helpful study advice, etc. 

1 2.6 Information Systems should be asked to conduct an initiation study to identify the feasibility 
and cost? of introducing the TRIX system. 

Evaluating student support 

12.7 It is recommended that a research project is undertaken to identify whether there is sufficent 
support for students, and if not, what steps can be taken to improve student support, and 
whether the role and recruitment of tutors needs to be changed to increase student support. It 
should look in particular at the roles that regional colleges might play in increasing local 
support for provincial distance education students. The project should involve a management 
team which would include a remote tutor, a representative of a regional college, a Programme 
Area Director or course co-ordinator, a remote student, someone from the registry, Jan 
Muirhead or Denise Hartmann, and a regional student advisor. It would conduct a survey of 
students and tutors, and would prepare financial estimates for any 
recommended changes. 

Validating assessment procedures 

12.8 It is recommended that, as well as monitoring by senior tutors, a means of analysing the 
reliability and validity of both continuous and examination assessment questions is devised, 
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to ensure consistency in the level of qualification. A small project team should be established, 
consisting of a programme area director or course coordinator, someone from the registry, 
someone from Information Systems, and an external consultant specialising in examination 
validity procedures, to recommend methods and procedures. 



12; Knowledge Network 

Knowledge Network requires a different approach from OLI in setting up a research and evaluation 
programme. OLI is more 'staff-intensive 1 than KN, and therefore OLI has more scope for 
evaluation and research to be incorporated into normal operational activities, and to some extent 
absorbed by current staff as part of their duties. 

KN on the other hand has very few staff, particularly on the educational side. These few staff are 
already heavily loaded, and even if they do take on evaluation and research responsibilities, their 
collective impact would still be small. They are therefore more dependent on external assistance for 
research and evaluation activities. 

The other major difference is that while OLI has good information on its students, and especially 
their names and addresses, KN •students' or viewers are either generally unknown, or available 
only through other institutions 1 registration systems. This makes the collection of information and 
feedback on programmes much more difficult, and hence more specialised and professionalised 
methods are necessary. 

Audience research 

While Nielsen and BBM statistics have their value, in terms of overall numbers watching KN, they 
have severe limitations. First of all, they are based on too few households to be statistically reliable 
with regard to the more specialised Knowledge Network audience, at least as far as individual 
programming is concerned. Secondly, they provide very limited information. As a consequence, 
key information, such as accurate, detailed and up-to-date figures of households both capable of 
and actually receiving specific Knowledge Network programmes, or viewers* reactions to 
programmes, is lacking. In order to provide this information, a number of steps need to be taken. 
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Recommendations 



13.1 A survey needs to be commissioned from a leading market research or census organisation 
(possibly Statistics Canada), to identify accurately, through careful sampling, Knowledge 
Network reception capability and utilisation within the province. If possible, the KN enquiry 
should be piggy-backed on a household survey already planned by the external research 
organisation, to reduce costs. The aim would be to identify across the province the number 
of households able to receive-KN-via cable with-and-without a convenor, or direct from the 
satellite (i.e. actual rather than potential reception facilities), the extent to which those 
households utilise KN programmes, and their reactions to the KN service in general. This 
will require extremely careful sampling procedures, and will be a relatively expensive 
exercise, involving several thousand households across the province. Data collected from the 
survey should be stored on the OLI computer, and c Elected in such a way that it is 
compatible with the OLI student data-base. This survey would not need to be repeated even' 
year, so should be treated as a project. 

13.2 The mailing list already held by KN, primarily for marketing purposes, could also be added 
to the market survey information, although it will be important that the two sets of 
information do not become confused, and work will need to be done to ensure that the 
marketing list is as compatible as possible with the OLA data-base. 

13.3 One benefit from the survey in 13.1 is that it could provide a good sampling base for 
enquiries about specific programmes, since from the initial (large) sample of people identified 
as able to receive KN programmes, smaller samples representative of the general population 
can be drawn for specific programming enquiries, and rotated to avoid over-use of certain 
households. 

Project research 

For reasons given in the introduction, research and evaluation at KN should be mainly in the form 
of development or research projects or special evaluation studies rather than the standard evaluation 
procedure recommended for OLI (at least until a significant proportion of KN's programming is in 
association with OLI courses). 
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Recommendations 



13.4 KN ought to have a greater sum for project research than the other programme areas, and 
make greater use of external research organisations. (The recommended minimum figure is 
$100,000). 

1 3.5 Professional development time should be used to allow KN staff to direct and participate in 
externally commissioned.projects. 

13.6 It is recommended that research be directed at key programming policy decisions over the 
next few years. For instance, research might be conducted into: 

(a) the effectiveness of live, inter-active programmes 

(b) appropriate production styles for different target audiences (e.g. general education, 
telecourses, teleseries) in terms of perceived audience need 

(c) the impact of video-recorders on the use of KN programming (access to recorders 
by different target audiences, implications for production style, support materials) 

(d) the need for media education (how to 'read' television) 

(e) ways to increase student learning from television (production style, use of recorders, 
signposting) related to different programming areas 

(f) better ways to identify and select appropriate programming (relative merits of market 
research, advisory committees, how to involve wider range of agencies, etc.) 

(g) better ways to liaise with wider public (a computerised directory of agencies, use of 
subscribers/mailing lists, newspaper, open-access video-boxes, for public to record 
opinions about needed programming and support services) 

(g) analysis of programming: sources, relcance to BC, spread of target groups, market 
research into demand for areas not covered, provision of programming for the 
disadvantaged 
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(h) reactions to imported programming 



Some of these studies could be integrated. Research methods would include postal 
questionnaires, telephone interviews, group discussions and individual, person-to-person 
interviews, using small but carefully selected samples of actual programme viewers. Some of 
these projects would be appropriate for joint research with Universities in the province. 

13.7 To ensure that research is directed to meeting the needs of KN, and is properly co-ordinated 
and integrated with programming and development work, it is recommended that a small 
committee of KN staff, plus the Director, Research and Evaluation Methods, be established 
to define research needs, allocate funds, commission studies, and disseminate and act on the 
results. 

13. Organising and funding reseaivh and evaluation 

Principles 

Throughout the consultancy, I have been guided by certain basic principles and assumptions: 

1. Evaluation and research should be the responsibility of each component, department, and 
operational unit. 

2. It is therefore the responsibility of managers to ensure it happens. 

3. Evaluation should be considered a normal part of everyone's work. Research on the other 
hand needs to be planned as a special activity. 

4. Evaluation should be considered as part of project management, and planned and resourced 
in exactly the same way as other project activities. 

5. Research studies *ould be treated as projects, and managed in the same way as other 
projects. 

6. Extra resources for evaluation and research activity would be limited, but nevertheless there 
may be some scope for extra resources be;ng found. 
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7. Otherwise, evaluation and research resources would need to be found either by reducing 
some other activities, or by increased efficiency. 

I have, by and large, been able to accommodate my proposals to these general principles. Virtually 
everyone interviewed recognised that evaluation and research ms. their responsibility within their 
own area, and they were willing to do research and evaluation. 

Making better use of people's time 

However, equally, almost everyone was concerned about lack of time and lack of expertise. 
The lackof time 

The lack of time can be met by re-organising people's workloads and a modest increase in 
resources. Time management is generally poor. Individuals should aim to plan their time across- 
various activities, and try to achieve their goals within that time. However, there appear to be no 
'norms* for various activities, such as required manpower (in days/months) for course design, nor 
any strict enforcement of production time for a course. This is a management responsibility: 
without such guidelines, it is impossible for individuals to plan their work efficiently. This is a 
larger issue than evaluation and research, but I believe that bett^ 'me management would free up 
some time for ew.' jation and research activities. 

Recommendations 

14. 1 (a) rough estimates should be made of person-days required to produce a course, across 
the various job categories (this is necessary in any case for proper project 
management) 

(b) each individual in the course design and production area should negotiate with their 
immediate manager a rough estimate of how time will be spread over various work 
activities; these estimates should take account of ail the demands on peopled 
work-times (course maintenance, administrative activities, evaluation, professional 
development, etc.) 

(c) commitment to new course production would take account of person resources 
available, and this would influence project planning; once committed, courses 
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should be produced according to the agreed schedule, and staff, having agreed the 
time necessary, would work to those norms 



The lack of expertise 

The problem of expertise is more difficult There is a professional body of knowledge regarding 
institutional research and evaluation, and a large number of skills required* I do not think it is 
feasible to bring everyone up to the necessary minimal standards by in-house seminars or by 
contracting in professional expertise on a short-term basis. Perhaps most important of all, good 
quality evaluation and research requires methods to be adapted to the organisation and practices of 
the system in which research and evaluation is to be conducted. Therefore there are strong reasons 
for requiring an internal, full-time specialist in this area, with both specialised knowledge of 
evaluation and research methodology, and a good internal knowledge of the system. 

I also believe that if limited resources are to be used efficiently for evaluati<M and research, there is 
a major need for co-ordination. Such a person would have Agency-wide responsibilities, and 
would play an important role in liaison and joint studies with other organisations in and outside the 
Agency. 

Lastly, a large number of people doubted that evaluation and research would happen to any extent, 
unless there was what the French call an 'animateur': someone who would encourage, advise, 
co-ordinate and motivate people to carry out the evaluation and research activities in an energetic 
and professional way. 

Recommendations 

14,2 It is recommended that a Director, Evaluation and Research Methods, should be appointed, 
with the following responsibilities: 

(a) advice to CEO and Executive Council on evaluation and research matters, including 
the feasibility and quality of research proposals and evaluation procedures within 
OLI/KN. 

(b) the design and contracting of evaluation and research activities commissioned by the 
Executive Council 
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(c) organisation and analysis of statistical data about students and courses for the 
Executive Council 

(d) advice to staff in OL1/KN and other B.C. open learning agencies on evaluation and 
research methodology, including advice on the design and conduct of research and 
evaluation activities, appropriate external research agencies, sources of funding, 
report writing, and dissemination ' - 

(e) the co-ordination of evaluation and research activities, to avoid duplication and to 
ensure widespread dissemination of research and evaluation findings within OU/KN 

(f) liaison with other open learning agencies involved with research and evaluation, 
with a special responsibility to disseminate research findings from other institutions 
of relevance to OLI/KN. 

14.3 The appointment of a Project Officer (Evaluation and Research), to assist OLI/KN staff in 
using the data-base, to draw samples, and to process and analyse research data; the project 
officer would be responsible to the Director, Evaluation and Research Methods 

14.4 Three options are offered regarding the organisational loc ' «i of the Director, Evaluation 
and Research Methods, and the Project Officer: 

i Directly reporting to the CEO, in the same way as other Authority-wide services, 
like the International Office 

ii Location within the Administration component 

iii Creation of a new component (Authority Services) which would combine all the 
Authority-wide services, including evaluation and research assistance 

My own preference is for (i) in the short-term, but (iii) in the long term. I am not happy 
about (ii), because the Director should be independent of operational units. 

Funding 

14.5 (a) The Executive Council should set priorities for research or special evaluation 
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projects requiring more than $50,000 in resources (manpower and/or cash, 
including a cost for time of existing staff assigned to work on the research project); 



(b) Vice-Principles should be free to approve evaluation and research activities within 
their own area costing less than $50,000 per project, provided they have the 
resources within their areas, and have checked with the Director, Evaluation and 
Research Methods, regarding the feasibility and quality of the proposals and lack of 
duplication with activities in other areas. 

14.6 Standard evaluations should not require extra resource. The procedure recommended is not 
labour-intensive and should reduce the overall rime spent on changes to course material. 

1 4.7 If the recommendation regarding professional development time is accepted (recommendation 
5), this will provide a substantial pool for approved development, special evaluation and 
research projects, although development, evaluation and research will not be the only 
activities carried out in professional development time- There could though be perhaps a drop 
in course production of around 5-8% per year (maximum) to accommodate this 
recommendation, even allowing for some efficiency savings from the standard evaluation 
procedure and better time management, unless extra resources are found. 

14.8 This leaves three options with regard to the recommendation regarding professional 
development time: 

(a) don't do it 

(b) see it as a replacement activity for some other activities 

(c) cost it as an extra resource requirement 

14.9 Revised procedures for making the data-base more accessible arJ purchase of appropriate 
software should be considered part of Information Systems normal development budget. If 
though other demands of equally high priority mean that these revisions wou? J be 
delayed, e::tra resources (probably not in excess of $60,000), should be found from the 
1987/88 budget. 

14.10 In addition to a slight reduction in course production activities, these recommendations 
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require the following "extra resources per annum: 

Salary and overheads for Director, Evaluation and Research Methods $60,000 (?) 
Salary^ and overheads for Project Officer (Evaluation and Research) $ 40 ,000 (?) 

Cash for special evaluation and research projects: 

Executive Council $m m 

Universit y $50,000 

0penCoUege $50,000 
Administration $50 000 

Knowledge Network $100 000 

Total 

$450,000 

Expenditure of £450,000 should enable all the proposals and projects suggested to be implemented. 

Implementation of proposals 

Some thought needs to be given about the implementation of the proposals that are accept-d. It 
might help to indicate the sequence and relatedness of the different proposals. The following is the 
recommended sequence. 

L Alteration to the data-has* ^rtVT IS priority 

If not implemented, most of the other proposals cannotbe implemented, 
either. 

2 - Appointment nf Director arvj project Off^r $100,000 
Essential for advice on standard evaluation, project research, 
and supporting evaluation activities 

3. Tuto r evaluation Nil cost 

Could begin immediately, if senior tutors agree 

These three proposals could be initiated immediately 
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4. Standard evaluation procedure put into place Nil cost 

Dependent on establishment of project management and data-base changes 



5. Project research >$350,000 
Some development projects could not wait, but major research 
projects should await appointment of Director and establishment 
of data-base requirements 

The implementation of proposals approved by the Executive Council will be the responsibility of 
the various programme areas. It might though be worth considering who is to be responsible for 
ensuring that the proposals accepted by the Executive Council are implemented, since 
implementation is likely to require some considerable effort and re-organisation. If the appointment 
of a Director is approved, that person could be responsible for reporting to the Executive Council. 
If not, the Council should nominate someone else to monitor progress. 

It should be possible though for the majority of the proposals to be implemented and operational by 
May, 1988, although some of the research projects may need to be phased in over a longer period, 
if $450,000 is not available in 1987/88 for these proposals. 
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INSTITUTIONAL RESEARCH AND EVALUATION CONSULTANCY 

TERMS OF REFERENCE 



1. TO DEFINE THE INSTITUTIONAL RESEARCH NEEDS OF THE OPEN 
LEARNING AUTHORITY 

2. TO ASSESS THE RESOURCES LIKELY TO BE AVAILABLE FOR 
INSTITUTIONAL RESEARCH ACTIVITIES (INCLUDING EXISTING 
MANPOWER) AFTER DISCUSSION WITH APPROPRIATE UNITS IN OPEN 
LEARNING INSTITUTE/KNOWLEDGE NETWORK 

3. TO DRAW UP AN INSTITUTIONAL RESEARCH PLAN, WHICH WILL INCLUDE 
WITHIN THE RESOURCES AVAILABLE APPROPRIATE: 

(i) INSTITUTIONAL RESEARCH ACTIVITIES (INCLUDING 
PRIORITIES) 

(ii) ORGANISATIONAL ST2UCTURE(S) 

(In) STAFFING (INCLUDING JOB DEFINITIONS), RESOURCES AND 
TRAINING IN INSTITUTIONAL RESEARCH 

(iv) METHODS OF CONDUCTING INSTITUTIONAL RESEARCH 
ACTIVITIES 

(v) SYSTEMS OF DISSEMINATION AND FOLLOW-UP 

4. TO DISCUSS AND WHERE POSSIBLE AGREE TO THE PROPOSALS IN THE 
INSTITUTIONAL RESEARCH PLAN WITH ALL CONCERNED UNITS IN EACH 
RELEVANT INSTITUTION BEFORE FINAL PRESENTATION OF THE PLAN 

5. TO PRESENT THE INSTITUTIONAL RESEARCH PLAN TO THE EXECUTIVE 
COUNCIL FOR AMENDMENT, APPROVAL AND IMPLEMENTATION 

6. TO BE AVAILABLE FOR CONSULTATION TO THE 'WING' COMMITTEE ON 
RESEARCH, TRAINING AND PROFESSIONAL DEVELOPMENT IN OPEN 
LEARNING 

7. TO BE AVAILABLE FOR CONSULTATION ON OTHER OPEN LEARNING 
AUTHORITY ACTIVITIES, UP TO A MAXIMUM OF 12 DAYS IN TOTAL 
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Definitions 



Institutional research: research which aims to improve the 

effectiveness and efficiency of open learning, 
through changes in teaching, administrative 
and operational activities, generally based on 
the systematic monitoring of existing methods 
and practices. 

Development: the introduction of new methods and practices 

for open learning (including applications of new 
technology) and the provision of externally 
contracted services other than distance 
education courses to other organisations. 

Professional development activities undertaken by individuals to improve 

their professional competence, whether 
instigated by themselves or their employer. 

The consultancy has been restricted by agreement to: 

(a) institutional research, and not development, except where the 
organisation of institutional research might overlap with 
development activities. 

(b) advice to the 'Wing' committee on the provision of research, 
professional training and development in open learning, for 
those both within end outside the Open Learning Authority. 

(c) consultancy on other Open Learning Authority activities, up to a 
maximum of 12 days 

Consultant 

The consultant appointed is Dr. A. W. Bates, Professor of Educational Media 
Research, and formerly Head of Institutional Research, at the British Open 
University. He will be working with the Open Learning Authority from 
January 5th until April 3rd, 1987. 
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INDIVIDUALS INTERVIEWED 

Institute Knowled g e Network 



Open Learning Instituti 

Glen Ferrell 
Ion Mugridge 
Dick Scales 
Sid Segal 
Shannon Timmers 
Mike Battistel 
John Bottomley 
Norms Macovi 
Jocelyn Calvert 
Denus Meekin 
Walter Uegam? 
Doug Cronk 
hark Neilsen 
Maureen Lewreres 
Hcnonne Loader 
Roseme r y Cumnghes r ' 
Dan Power 
Kerel Bare;*': 
Susan Lesiie 
Susan HaQ'und 
Diane Read 
Jan Muir'nead 
Heather Drucge 
Denus Meekin 
Tony Lil'iy 

Mary Anne Williamson 
Deb Baker 
Dave Gilmore 
Peter Dailiy 
David Symington 
Don Weir 
Hardi' Fische - ' 
Care! Gib^C'*" 



Lucille Pacey 
Alexandra MacGregor 
Gresham Bradley 
Mike Reddington 
Ron Harrington 
Megan Carvel 
Betty Mitchell 
David Roach 
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Appendix B 



GROUP INTERVIEWS 

University course designers 
CTV course designers 
University senior tutors 
CTV regular tutors 

PROPOSALS CIRCULATED FOR COMMENT TO: 

All those individually interviewed (as relevant) plus: 

Bruce Thompson 
Penny Street 
Catherine Kerr 
Bob Amussen 
Linda Love 
Jim Wnght 
John Moore 
Carol Renr:? 

lily Liu 

Ja^e Anderser* 
Lois Harries 
Joan Richest 
G r 6 n Dc* ) 1rnc , *~ r ' 



i OLI Staff Publications Held In The Archives of 
The Open Learning Institute Library ~~ 



Bates, P. Marketing at the Open Learning Institute: a status report . 
Unpubl i shed report. The Open Learning Institute, 1986. 

Bates, Patricia. "VAX offers resource management system for distance 
education." EDU No. 35 (Spring/Summer 1984), 16. 



Battistel, CM. & Ed Giovanella. "Access control and utilization monitor." 
RSTS Professional 3:1 (1981), 74-78. 



Bottomley, John. Discussion of the adaptation of British Open University 
materials for Open Learning Institute Courses" Unpublished paper. 
The Open Learning Institute, 1979. 

. Open Learning Institute enrollment projections 1979-1984 . Unpublished 
report"! The Open Learning Institute, 1979. 

. Print production audio production and materials distribution at the Open 
Learning Institute: a status report . Unpublished report. The Open Learni 
Institute, 1986. 

. "Production, storage and distribution." in Distance education in Canada , 
ed. by Ian Mugridge and David Kaufman. London! Crovn Helm, 1986. 



. "The Production of distance education materials: an industrial 
process?" Symposium Paper presented at the American Educational Research 
Association Annual Meeting, New York, March 1982. 



__. "Television and satellite use at the Open Learning Institute." 
Unpublished paper. The Open Learning Institute, 1980. 

Calvert, Jocelyn. "Delivery and support of science distance education." 
in Abstracts of the 15th Pacific Science Congress, Dunedin, New Zealand, 



. "Facilitating transfer of distance courses." Open Learning 1:1, 
(1986), 34-37. 

. "A perspective on distance education research." Paper presented at 
Working Group Meeting on Distance Education Research, Universiti Sains 
Malaysia, 1984. 
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: . "Research in Canadian distance education." in Distance Education 

in Canada . ed. by Ian Mugridge and David Kaufman. London: Croom Helm, 



. "Science degree programmes at the Open Learning Institute." 
Unpublished paper. The Open Learning Institute, 1985. 



. "Tutoring in the most industrialized form of education." Paper 
presented at the annual convention of the American Educational Research 
Association, New York, 1982. 



Cunningham, Rosemary. "Distance education in Canada." OLS Nevs 13 (1984), 6. 



Dave, Alan. "Planning for adult basic education." Unpublished paper. The 
Open Learning Institute, 1979. 



Durvard, G.W., & M.L- Durvard. A telephone survey of distance education 
students at the Open Learning Institute . Unpublished report. The Open 
Learning Institute, 1983. 



Ellis, John. "The role of research in the development of the procedures 
and programs of a comprehensive distance education institute." Paper 
presented to the 21st Adult Education Research Conference, Vancouver, 
British Columbia, May 1980. 



Ellis, John F. and John Bottomley. "Problems and opportunities associated 

with developing and operating a distance education system. 1 ' Paper presented 
to the Open University Conference on the education of adults at a distance, 
Birmingham, U.K., Nov. 18-24, 1979. 



Ellis, John F. and I. Mugridge. The Open Learning Institute of British 
Columbia* a case study . Distance Education research Group Papers 
Number 8. The Open University, 1983. 



Farrell, Glen M. and Margaret Haughey. "The future of open learning", in 
Distance Education in Canada , ed. by Ian Mugridge and David Kaufman. 
London: Croom Helm, 1986. 



Formative evaluation report, Spring semester, 1980 . Unpublished report. 
The Open Learning Institute, 1980. 
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Forsythe, Kathleen. "Knowledge ' \ new hybrid for learning systems." 

Distance Education 3:2 (1982)^ 



• "Open skies, grass roots." Unpublished paper. Victoria, B.C.: The 
"Knowledge Network, (19871. 



Forsythe, Kathleen, and Valerie Collins. "British Columbia- higher education 
and the integration of a nev technology." A case study prepared for Project 
It Area B, Cnetre for Educational Research and Development, OECD, November, 
1983. {unpublished?] 



"Understanding the effectiveness of media in the learning process." 
Paper presented at the World Congress on Education & Technology, Vancouver, 
May, 1986. 



Grayson. John. "Continuing education: a proposed policy and options for 
involvement." Unpublished paper. The Open Learning Institute, 1979. 



s The use of audio and related media within the communicational and 
operational context of three distance learning institutions; a se~of 
options and recommendations for the use of audio and related media in 
distance learning ] Unpublished report. The Open Learning Institute, 1979. 



Kaufman, David. "A computer-based instructional system for distance 
education." Computer Education 8:4 (1984), 479-484. 



* "Course development: industrial or social process." Paper presented 
at the Annual Meeting of the American Educational Research Association, 
Nev York, March 1982. 



. "Distance learning systems for training." Unpublished paper. The 
Open Learning Institute, 1983. 



. "The future of computer-aided learning at the Open Learning Institute: 
a proposal ..." Unpublished paper. The Open Learning Institute, 1984. 



. "Practice and theory of distance education: course blueprint." 
Distance Education 5 (1984), 239-251. 



. "Programs of study at the Open Learning Institute." Summary of 
presentation given on May 27, 1982 at the Distance Education Conference, 
Montreal Que., May 25-28, 1982. 
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Kaufman, David, and John Bottomley. "The Opeh Learning Institute: entering 
the twenty-first century [Seminar session presentation]" in Compromise 
collaboration or competition in higher education ; Proceed i ngs of the 
Forty-first Annual Pacific Northwest Conference on higher Education, 
Vancouver, B.C., October 10-12, 1979. Corvaliis OR: Oregon State 
University Press, 1982. 

. "The Open Learning Institute: university program development. Steering 

Committee for the International Institute for Distance Learning." 
Information Unit Newsletter No. 7 (1930), 12-27. 

Kaufman, David, and Denise Sketches. "Designing visuals in self-instruction." 
Paper presented at the 14th Annual Meeting of :he International Visual 
Literacy Association, Vancouver, B.C., Nov. +o t 1982. 

Kaufman, David and Robert Sweet. "Increasing educational opportunity for 
adults: a Canadian example." Higher Education in Europe 2:3 (1983),?. 

Kaufman, David; Usukawa, Saeko; and Denise Sketches. "Visual design in 
distance education." Canadian Journal of Educational Communication 12:2 
(1982), 4-13. : 

Le Couteur, P. The feasibility of methods of offering a distance learning 
chemistry course in British Columbia . Unpublished report . The Open 
Learning Institute, 1982. 

Love, Linda. "Guidelines for planning and conducting a teleconference." 
Unpublished paper. The Open Learning Institute, 1983. 

. The Telidon Project Report . Unpublished report. The Open Learning 

Institute. 1982. 



Macey, Sam. "British Columbia's Open Learning Institute." Management Servic 
27:8 (1983), 20-25. 

Mclnnes-Rankin, Ethelyn. Adult student and tutor perceptions of academic 
advising at the Open Learning Institute . M.A. Thesis. Vancouver, B.C.: 
University of British Columbia, 1986. 

Meakin, Denys. "A flexible registration course length and examination system 
for distance education." Paper presented at the ICDE Conference, Melbourne, 
August, 1985. 
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_ . "The role of regional centres." in Learning at a Distance , ed. by 

J.S. Daniel, M.A. Stroud & J.R. Thompson"! Edmonton, Alberta: Athabasca 

University/International Council for Correspondence Education, 1982. 

. "Student services for lightly populated regions." Paper presented to 

University and College Library Section, Adelaide, 23 August, 1985. 

"Use of statistics." Paper presented to Management of Distance Education 
Special Interest Group, ICDE Conference, Melbourne, August 1985* 

Mugridge, Ian. "Applications of computer technology in distance education: 
the case of the Open Learning Institute." Canadian Journal of Educational 
Communication 14:1 .(1985), 6-7.. 

. "Consortia in distance education: some Canadian ventures." Open 
Campus 8 (1983), 22-29. 

. "Distance education and open learning in British Columbia." a.c.u. 

Bulletin of Current Documentation 76 (Dec. 1986), 2-4. 

. "Distance education in British Columbia." ICDE Bulletin 13(1987), 

46-49. 

"Distance education- a major role to play in the Third World." IPO 
•Nevslett^7:3 (Un'er 1985-86), 10-12. 

. "The establishment of a new distance education institution: the Open 

Learning Institute of British Columbia." Distance Education 2:1 (1981), 
98-109. 



. "The Open Learning Institute." in Distance Education in Canada , ed. by 
Ian Mugridge and David Kaufman. London: Croom Helm, 1986. 

Mugridge, Ian, and David Kaufman eds. Distance education in Canada > London: 
Croom Helm, 1986. 

Paterson, J.C. "Student advisory service: regional centres-functions & 
staffing." Unpublished paper. Th* Open Learning Institute, 1979. 

Rothe, J. P. "Audio teleconferencing and distance education: towards a 
conceptual synthesis." Distance Education 6 (1985), 199-208. 
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Scales, K. "A study of the relationship between telephone contact and 
persistence." Distance Education 5 (1984), 268-276. 



"A typology applied to distance education in British Columbia." 
Lifelong Learning 7 (1983), 26-28. 



Slade, S., and B. Webb. "Canadian off-campus library services survey." 
Unpublished report. University of Victoria, 1985. 



Sweet, Robert. "Applying Tinto's model of student drop-out to distance 
education." Unpublished paper? The open Learning Institute, 1984. 

"Distance Education: the personal response." Paper presented at the 
Annual Meeting of the American Educational Research Association, New York, 
March 19-23, 1982. ED 216 658. 



"Student attrition at OLI: student views of the attrition process and 
profile of student characteristics." Unpublished report. Open Learning 
Institute, 1981. 



Summative evaluation report & student profile summary. Summer semester 1980. 
Unpublished report. The Open Learning Institute, 1980. 



Timmers, S.F. "Developing biology courses for distant students." in Abstracts 
of the 15th Pacific Science Congress, Dunedin, New Zealand, 1983. 



"Distance education- over the rainbow." EDU no. 35 (Spring/Summer 1984), 
^.10. 



"Microcomputers in course development . " Programmed Learning and 
Educational Technology 28:1 (1986), 15-23. 



Timmers, S.F., and J. Calvert. "Science distance education at the 15th Pacific 
Congress, New Zealand." ICDE Bulletin 3 (1983), 46. 



Timmers, S.F., and I. Mugridge. "The Open Learning Institute: recent 

innovations." in International Yearbook of Educational and Instructional 
Technology 1986/87 . ed. by C Osborne. London and New York: Kogan Page: 
Nichols, 1986. 
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2. PUBLICATIONS ON THE OPEN LEARNING INSTITUTE BY EXTERNAL 
WRITERS HELD IN THE ARCHIVES OF THE OLI LIBRARY 

Brookf ield, S.D. and Harvey, L.N. ( 1 98 1 ) Semester Enrollment and 

Examination Structures at the Open Learning Institute: Alternative 
Perspectives (commissioned paper.) 

Cherry, J. ( 1 984) Planning for Lifelong, Adult Education at Britain's Open 
University and British Columbia's Open Learning Institute with 
reference to the recommendations of the Faure Report. UBC: Dept. of 
Adult Education (student assignment) 

Heselton, N.6. (19S5) The Future of Teleconferencing at British-Columbia s 
O pen Learnin g In stitute. Simon Fraser University (MBA thesis) 

Woodley, A. (1983) Institutional Research and Development at the Pper 
Learnin g i"?*.:yj<? UK Open University (commissioned paper) 
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"Car.auian Journal of University Con'tinuin^ Education, Vol* 
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%% 7h«* u£ V plo:r.ent of the Knowledge .\etwork'« t Chemistry in 
the Eigntles Cor.f^r^nce, Cir.cn Fraser University* riovfrcer, 
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^;Leernir.- in the Eighties'', College Canaca. Vol. £ fco. 6t 
jan/Fec Scrir.^ 19S2 (witr. \. Forsythe). 

**The *ay for a Scatl CoUece to Get Further raster", in 
"CctprorisA, Collaboration or Competition in Higher Ecuca- 
; tion<>, Oregon State University Press, (with Sal- 

ter) . 
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/'Data Uses of the Inter-Ins tltuticnal Service of the 
jKnowteupe f,et«orM », August,, 1962. 
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* % An In trocuc t i ON tc Open Systems Architecture and Broac- 
ranc -\V tvorry'^r P^-fr-ncr-l e trrco n tu kuiI Cunfn ci;ip, J ♦ 5 . C ♦ 
{Aucust , 1 • 

!* % Cotcutus and Strat^j/ - J Design Report for the Develop- 
-lent one Deliver* of C ou r se» or e 1 » , FeLrnary, 1953 (with w.3. 
jOocfrcy et al). 



| %% Towarcs a Corr.put^r Fluent Populace'', Fcrtr. Canadian Syrr- 

joosiu.T. on Instructional Technology, winni^ec, Cctoner, 19d3 
j<*ithK. Forsytne). 

T'Corputer »#e t ko r * <: 1 ' , Kno».l ecve Network ,\ew;.. Fall, 1933. 



f "»uRlic >t'wcris r Myth or Reality", In t er-Pr o v in ci a t 
Association for Telematics and Telicon Seninar, Victoria, 

r * '• i - r Technical - e'M/i reT»*nt s of Lin>in.: Learning 
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roller les Institutions ecucatlonnelles du Canada a.vec Les 
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tion inter-p rovt nc 1a Le de telemattcue et de Telicon f Octobre 
1984. 

* % The Sucessful Establishment of a Prot /pe Network » Inning 
^Ca naolan Learn Ing Inst 1tut tons and Educational Databases 1n 
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5 0 R T E L» April 1585. 

Teleconferencing by Computer - An Introcuct icn« » f CONS 
OR T £ Lt April I9e5. 
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Electronic Pub 1 1 shine , August , 1985. 
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•* J -£l a s~c cr ha Ii.fui ii.u I i ca — en la e uulj i \ u n — el Canada ' 

Pr Irer Cone re so Col ombiano de Informatlca, Cartacena, 
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''Teaching Computer Science 1n China by CAL and Telecon- 
ferencing**, CIPS Congress 66, Vancouver, 1966. 

%w The Gold Coin Consortium of Colleges 1n Ontario: From Con- 
cept to Transf ormatlon** , 5th Canadian Symposium on Instruc- 
tional Technology, Ottawa, 1986 — — ~ - 

'*Ket*orKS and Education 1 *, World Congress on Ecucatlonal 
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*'Tfc* Impact o"f Machine Indeoencent Software on Higher 
Ec t1on»», Communications Technology 1n Higher Eoucation 
Conference, M oncton* 1966. 
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lcpologies" , University of *aterl< , Networking Conference, 
Waterloo, 19e6. 

% 1 Te lecommun 1c at 1op. Strategies for Open Learning' ' , 
Knowledge Network, 1986 (with R.J. Martin anc D.«. Reach). 



^'Teaching at a Olstance with CAL»', Open Learning, Issue 3* 
1966 (with h.O. Godf rey ) . 

"froject Clear SMes'S Computer Journal (British Computer 
Society), In press. 
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OPEN LEARNING INSTITUTE (1982) Self -Study Repo rt. 
OL1 Self-Study Steering Committee 

KEEGAN, D. (1982) Student Support Services at OLI 
Richmond: OLI 
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OPEN LEARNING INSTITUTE (1982) Report of External Team 
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MEMORANDUM 



From: Tony Bates, IET 



Ext no 313 



To: Peter Dailly, c.c Dennis Meakin 



Room no: hors de Heron 



Subjact: Improving the data-base 



DMe: 12 February, 1987 



Problem. The data-ba^e is not organised at the moment in suchavay that basic 
information about courses or students required for evaluation purposes can be 
readily abstracted. Much of the information required can be obtained through using 
Datatrieve, but it requires extremely laborious procedures and a great deal of 
knovledge, both of the data-base and the Datatrieve programme. Other information 
is in fact impossible to abstract in the form necessary. This limits information about 
basic performance indicators to a very small number of people, all of vhomhave 
other, more pressing operational priorities. Unless this bottleneck is removed, 
evaluation of courses on a systematic and reliable basis is not possible. 

As an example, I attach an enquiry from Jocelyn Calvert regarding drop-outs. After 
discussion vith Heather Drugge, it became apparent thatit vould take 40 hours of 
Heather's time to get this information. Despite the fact that I am an experienced 
researcher, snd familiar vith data-bases, it vould have taken me considerably 
longer. This is not practical, yet the enquiry is a reasonable one, and the information 
required for the enquiry is actually in the data-base. 

Cause of the problem. Tfca problem is caused because individual student files are not 
complete. I understand that the data-base is constructed as follovs (please correct 
me if I am vrong). There are three file systems. The master student file is vhat is 
entered once only from the initial registration form Then there is a master course 
file for each course, vhich lists details of the course, but noj student information A 
third file is then created (the student course file) for an individual course, vhich 
takes information from the master course file, 'reads' the student files, then 
identifies vhich course the student is registering for, then transfers the student 
numbers onl y to the student course file. This leaves important demographic 
information in the student master file. To get any other information about students 
on any course requires special programmes to be vritten, or the use of Datatrieve. 
In other vords, to match student performance vith other student information, a 
special record has to be created for each student each time. 

jReq uirement: W hat is required is a programme vhich automatically merges nev 
student data vith existing student data, in a consistent vay for all students. Thus 
after a student registers for a course, not only should course performance 
information (e.g. grades) be added to the master student course file, it should also 
automatically be added to the master student file. Similarly, vhen a student registers 
Jor a second (or subsequent) course, all the student information regarding the 
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second course should be added to the initial master student file. All fields on the file 
should be keyed fields. Thus ore is creating an easily accessible, dynamic, 
continually up-dated and above all complete record for each student. Once such a 
file is created for all students, this becomes the master data-base from vhich all 
subsequent analyse can be done for evaluation and research purposes. footer 
vords, the computer vill search through the master student file according to 
parameters set by the enquirer - for instance "Select all students on courses vith 
ADMN codes for all sessions, vho previously studied an OLA pre-umversity 
course". This vould then provide the sample on vhich further analyses can be done. 
Using this system, itshould be feasible to abstract appropriate sample information 
for local processing on micros, tin* reducing CPU time. (It may also be possible to 
get rid of the master student-course file altogether.) 

Note that vhile this is a requirement for evaluation and research purposes, it vill 
also become increasingly necessary as more and more students complete degrees, or 
seek credit transfer, since a full student record vill be required. Also, some 
countries nov require, undei public data pre 'action acts, that all stored information 
about an individual should be available to that individual on request. My proposal 
vould make this a much simpler process. 

Once this basic master student file is created, standard statistical analysis packets 
(such as SPSS) can be bought, vhich vill enable data to be easily abstracted in the 
form required by anyone authorised to do so, vhether or not they have 
programming skills . 

Pfrfflprnmnriprf m-Mmi Information Systems should be requested to do an initiation 
study to see vhat is required to provide a basic, dynamic master student file, and 
vhat packages vould be most appropriate for abstracting and analysing data, 
including standard statistical tests. 



TO Tony Bates 

FROM Jocelyn Calvert 

RE A datarrieve project 

DATE 22 January 1986 



lev. 



I'll describe two of a million possible projects and give some 
background for each. 

Course completion over gore than one registration period 

Objective: To determine what proportion of students receiving an 
incomplete (I) grade during their first registration subsequently 
complete the course. 

Rationale: Students who do not complete a course during the normal 
time period are permitted to re-register at a reduced fee and, on the 
recommendation of their tutors, to carry over assignment marks from 
their first registrations. (It is my opinion that they are also 
permitted to carry over passing exam marks, but I believe the registry 
dissents.) As a result, the completion rate for a session may not 
tell us the dropout rate. 

Possible strategy: / * ^ ^ * v I 

1. Select all students who registered in university courses for the 
first time in 8509 and who received an I grade. Such students may 
- have had a previous registration in a different course. 

Z. Determine what proportion completed the course with a grade of A, 
B, C or D in a subsequent session. A 

Notes: tf J 

This will not provide a complete answer because I think ve attract 
different groups at different times of the year. For example, the 
summer calculus crowd could result in different figures for May. 

Since the computer flags students who are re-registering in a course, 
I assume it can select those who are not. 

An overall figure for university courses would be useful to begin 
with. At some point we might want a bieakdown by course or the same 
information for a single course. 
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MEMORANDUM 

From: Tony Botes Ext no.: 3 1 3 

To: Jocelyn/Heather Room no: Outside Heron 

Subject: Dototrieve exercise Dote: 26 January, 1987 

How many students who foil or withdraw go on to complete in 
another session? 

Step 1 Take one session (8505) and discover what kinds of students in 
university programme pass, fail or incomplete (all students) 

8505 

All registrants for 01) courses in 8505 (A11=e) 

Tohle t. hyored 



Pass 


low.main 
No. % 


Other urban 
No. % 


Pur?] 1 
No. % 1 


Tptol 
No. % 
b 


Fail 








c 


Incomplete 








d 


Withdrew 








e 


Total M 


100 


100 


100 i 


a 100 



Total e 2: by educdtion 

Codes 1-2 3 4-5 6,7 ,8,9 11 10 Total 

>6rode 1 1 HighS. Un.tr. Non-U.q. >Degree Degree 

No. % No. % No. % No. % No. % No. % No. % 

Pass 



Foil 

Incomplete 
Withdrew 

1 

Total (N) 100 100 100 100 100 100 100 
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Table J: by occupation 
Table 4- by course objective 
Tob/e !i by sex 

Step 2 Identify those who registered for the first time for each 

specific course, across oil university programmes (oil first 
time registrants = b) and identify their characteristics 

8505 

Students registered for the first time for o porticulor course 
Tob/e &• Jto/ times registered for course 

Pass Fail Incomplete I Total 

IN % 

First time I p 100 

I 

Previously registered I 100 
, 

Total (N) I a 

Step 3 Identify first time registrants who (a) failed (= c students) or (b) 
incompleted (= d students), and analyse by area, education, occupation, ond 
course objective. 

8505 

First time registrants who failed or incompleted 
Tob/e ?: by area 

Lower main. Other urban Rural Total 

I N % 

Pass 60 30 10 I 100 

I 

Fail 40 40 20 i c 100 

I 

Incomplete 30 30 40 I d 100 

_^ , 

Total (N) I b 



Table 6: by education 



Table 9: by occupation 



Tob/e 10: by course objective 



ten 4 Identify subsequent fate of first time registrants who failed or 



9 

ERIC 



Incompleted 

1805 

First time registrants who foiled or completed ond re-registered 
Table 1 1: by subsequent registratiaa(Hoie change of X column) 
Ra-reglstered for course Foiled Incomplete Iflifll 

8506 
8507 

8508-8604 
After 12 months 



Sub-total e f g 

Registered for other OU course 
Registered for other OLI course 
Not re-registered 



Total c 100* d 100% 

Table 12: by no. of re-registered sessions 

Re-registered for course Foiled incomplete Total 

Once 

Twice 

More 



Sub-totol 

Registered for other OU course 
Registered for other OLI course 
Not re-registered 



Total c 100* d 100* 

Table 15: by grades on re-registration (based on lost or best grade) 

Failed Incomplete Total 

A 
B 
C 
D 
E 



Sub-total m n p 

F 

Incomplete 

Registered for other OU course 
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Registered for other OLI course 
Not re-registered 



Total c 100* d 100* 

p = students who foiled or incompleted on their first session who then 
went on eventually to complete 

2ifiP_5 Identify characteristics of those who went on eventually to 
complete, combined with those who succeeded first time 



MEMORANDUM 



TO: OLI and KN Staff 



FROM: Ian Mugridge 

RE: Workshops on institutional research 
DATE: February 2, 1987 



As you know, Tony Bates of the British Open University is 
spending three months looking at the research needs of the 
Open Learning Authority. As part of his activities, he 
has offered to give a series of workshoos for institute 
and network staff. These have been scheduled as follows: 

(1) 12th February Creating a framework for IE 

- IR and decision-making 

HE5GN ROOM ~ c ^ n - position ~ 
qi^ t - the political context 

- IR ana development 

(2 } I9th February Collecting feedback on courses 

BOARDROOM 
OLI 

(2) 2&th February Designing project evaluation 
NETWORK 

( ^ ) 5th March Using tne data-base for 

HZEGK ROOK policy decisions 
OLI 

(5) 12th Karen Evaluating television and other 



KNOWLEDGE 
NETWORK 



technologies (at KN) 



/ 19tn Karch A plan for institutional research 

BOARDROOM 
OLI 



Each seminar will take place from noon until 1:30 on the 
days indicated. Coffee will be provided. 




cc . : Tony Bates yy 

OPEN LEARNING Ifl^TITUTE 



C-t *y A p fH 



£bo£Sc: ISC in Sci^/oW 8 7oj 
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